Cover Page

Copyright © 2003 by Microsoft Corporation


LOC

PUBLISHED BY
Microsoft Press
A Division of Microsoft Corporation
One Microsoft Way
Redmond, Washington 98052-6399

Copyright © 2003 by Microsoft Corporation

All rights reserved. No part of the contents of this book may be reproduced or transmitted in any form or by any means without the written permission of the publisher.
Library of Congress Cataloging-in-Publication Data
Developing International Software / Dr. International-- 2nd ed.
        p. cm.
     Includes index.
     Rev. Ed of:  Developing international software for Windows 95 and Windows NT/
Nadine Kano, c1995.
     ISBN 0-7356-1583-7
     1. Computer software--Development. 2. Microsoft Windows (Computer file) I. Kano,
   Nadine, 1968- Developing international software for Windows 95 and Windows NT. II.
   Dr. International (Group) III. Microsoft Corporation.

   QA76.76.D47 D479      2002
   005.1--dc21                                                 2002070303

Printed and bound in the United States of America.

1 2 3 4 5 6 7 8 9 QWT 7 6 5 4 3 2

Distributed in Canada by H.B. Fenn and Company Ltd.

A CIP catalogue record for this book is available from the British Library.

Microsoft Press books are available through booksellers and distributors worldwide. For further informa-tion about international editions, contact your local Microsoft Corporation office or contact Microsoft Press International directly at fax (425) 936-7329. Visit our Web site at www.microsoft.com/mspress. Send comments to mspinput@microsoft.com.

Microsoft, the Microsoft Internet Explorer logo, Microsoft Press, MSDN, the .NET logo, Office, OpenType, Visual Basic, Visual C++, Visual Studio, Win32, Windows, and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other product and company names mentioned herein may be the trademarks of their respective owners.

The example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious. No association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

Acquisitions Editors: Juliana Aldous Atkinson
Project Editor: Kurt Stephan


Foreword

I completed work on the first edition of Developing International Software in December of 1994. Eight years later, I still receive scattered e-mail with questions about running or developing world-ready software, which I usually forward to the very capable World-Ready Guides (WRG) team (who are also responsible for the Dr. International column on http://www.microsoft.com/globaldev) inside the Microsoft Windows division. Since the most frequently asked question by far has been "When is a new book coming out?" I was thrilled when the WRG team picked up the enormous task of doing an update. The finished product is a much more comprehensive treatment of the subject matter, which is needed and welcome given the central relevance of Web-based software, a phenomenon that has emerged since we published the first edition. Basic techniques and knowledge of the Microsoft Win32 application programming interface (API) could help developers produce world-ready software in 1995, but in today's Internet-laden climate they seek to understand the role of other Microsoft products such as Microsoft Office, Microsoft Internet Information Services (IIS), Microsoft SQL, and Microsoft Visual Studio .NET, as well as industry standards such as Extensible Markup Language (XML). It is heartening to see how much these products and technologies have expanded their support for international functionality over the past several years.

International support in Windows also continues to mature. While the basic foundation-Unicode and the National Language Support API (NLS API)-remains very much the same, functionality has deepened with each new release to encompass new locales, new locale information, and more sophisticated text-layout and font capabilities. The book delves into these updates and lists detailed information in the significantly improved appendixes. I am also happy to see an entire section on testing. Given increasing requirements for product reliability and security, particularly for global firms, nothing could be more important than thorough and methodical software testing methods.

The software marketplace has grown increasingly global over the past eight years but, even so, the English language still dominates. Shortly after I joined Microsoft in 1989, the then vice president of Microsoft's international division threw a party celebrating the fact that sales of localized products had reached 50 percent of company revenues. Recent figures are closer to 60 percent. Considering overall revenue growth, that 10-percent shift constitutes millions of dollars, but I still wonder why the figure isn't closer to 75 percent or more. Certainly Microsoft and many other companies have come to understand that globalizing software shouldn't be a separate effort. Microsoft, for example, no longer has an international development division-each product organization treats world-readiness as a core development requirement. Here in Redmond, we even host international teams from Original Equipment Manufacturers (OEM) partners and international subsidiaries to ensure that we address world-ready requirements on a consistent and constant basis. The results of the technical and organizational integration have included achievements like simultaneous shipment of several language editions of Microsoft Windows XP. But I still think that the mindset of the English-speaking developer stays too close to the "at home and familiar." This has ramifications beyond world-readiness-in general, our industry needs to do a better job of living and breathing the world of our customers. The scenarios in which companies deploy software are much more complicated than developers realize when they imagine how their designs and algorithms can be applied.

It is my hope that this book will become not only an indispensable reference, but required reading for any software development organization that takes its customers' needs seriously. Although challenges like sharing data and mixing and matching localized applications with localized operating systems have become easier to manage, it is up to the development community to make global interaction much simpler to achieve and much more powerful in its uses. The tools that enable such interaction will continue to evolve, but even once the infrastructure is perfected, developers will still need to take proper advantage of it for businesses and consumers to reap the appropriate benefits. For so long, those of us who evangelize world-readiness have emphasized "making it work properly" and lessening the pain of getting there. Following the methods outlined in this book should make "it works" a given. As a developer, your energy will be better spent on the still underexplored arena of multicultural, multilingual software communities and the untapped opportunities they present. "It works" is not enough. "Now that's a brilliant idea that will change the way people do things" is a much more fetching proposition.

--Nadine Kano

Redmond, Washington


Acknowledgments

Writing a reference book that covers various areas and technologies, while remaining technically pertinent and accurate, requires the involvement of many people. This book has been put together and, in large part, written by the World-Ready Guides (WRG) team-David K. Hsu, Houman Pournasseh, Björn Rettig, Russell D. Rolfe, and Rostislav Shabalin-in the Microsoft Windows Division, who are also responsible for the Dr. International column on the Global Software Development Web site at http://www.microsoft.com/globaldev. That's why Dr. International was chosen to be the honorific author of this book. However, it must be pointed out that many people have extensively contributed to this book's realization.

First, we would like to thank the Windows Globalization Services team who made Windows the best platform for multilingual computing and demystified some of the latest globalization technologies for us. Special thanks to John McConnell and his team who helped us better understand the Microsoft .NET Framework globalization services, MLang, and Microsoft Layer for Unicode (MSLU); Mohammed El-Gammal for his help with the "Graphics Device Interface Plus (GDI+)" chapter of this book; and Edward Ye for his contribution to the Multilingual User Interface (MUI) content.

The Microsoft Visual Studio and the .NET Framework teams have done an outstanding job in releasing the latest version of this development tool suite and working environment earlier this year. Kathleen Carey, Mohamed Hassanin, François Liger, Marin Millar, and Achim Ruopp's continuous help and support were greatly appreciated. The .NET logo throughout the book is a footprint of their contribution.

Without Uniscribe, handling complex scripts would have never been as easy and staightforward as it is today. The Uniscribe team's, enlightened support and guidance regarding Uniscribe's implementation and functionality have been outstanding. Special thanks to Worachai Chaoweeraprasit, Tarek Mahmoud Sayed, and David C. Brown, the father of Uniscribe.

WRG works closely with Windows localization and localizability teams to foster internationalization inside Microsoft. These teams' efforts, from the actual translation of software all the way to creating pseudo-localized builds, have enabled Microsoft Windows XP to be released into 34 languages within a short period of time after the English version was released. Our special thanks to Mohamed Elgazzar, Jan Roelof Falkena, and Stefanie Schiller for their help with the pseudo-localization and localization chapters of this book. It's amazing to see how much font technology has changed during recent years. The Microsoft Typography team is today extensively involved in providing support for additional scripts. Fonts that this team creates are used across different products in Microsoft. Our thanks and appreciation to David Glenn, Apurva Joshi, Paul Nelson, and the rest of the team for their contribution to this book.

Program managers in Microsoft subsidiary offices across different countries play a key role in defining future designs as well as country-specific and locale-specific feature implementations. We would like to thank them for their input and feedback. Our special thanks to Shane Casserly, Neil Powell, and Dominic Smyth.

Also, we would like to thank F. Avery Bishop for his useful and relevant published articles that this book borrows, Françoise Botale-Hovivian for the chapter on content localizability, Fernando Caro from the SQL team, Kevin Gjerstad from the Microsoft Windows Text Services Framework (TSF) team, Chris Lovett for his work on the "Extensible Markup Language (XML)" chapter, Gwyneth Marshall from the Microsoft Internet Information Services (IIS) team, Chris Pratley from the Microsoft Office team, Mohamed Sadek for his review of the "Mirroring" chapter, Murray Sargent from the Rich Edit team, Chris Wendt for his contribution to the "Hypertext Markup Language (HTML)" chapter, Michael Kaplan for his indefatigable review work and inspiration for many good discussions, and of course Nadine Kano for writing the first edition of this book and inspiring us to write this update.

And finally, we would like to thank Susan Joly, our editor from TCCI; Amy Shear and Joel Panchot of Microsoft Press, our production artists; Juliana Aldous, our project manager at Microsoft Press; as well as Kurt Stephan, Melissa von Tschudi-Sutton, and Maureen Zimmerman from the Microsoft Press editorial team for their help, support, and patience.

Internationalization is a vast project that involves a large number of great developers, testers, and managers here at Microsoft. The World-Ready Guides only spread the word on internationalization requirements; the credit goes to those who actually implement them. Our sincere thanks to all of those not mentioned specifically by name.

-Dr. International


How to Use This Book

As its title suggests, Developing International Software is a reference book on software internationalization. It is written for a beginner-level to intermediate-level audience in the field of internationalization, though it is assumed that programmers, testers, and project managers are already proficient in their respective areas of expertise. From a platform standpoint, the book chiefly targets Microsoft Windows XP, as well as Microsoft Windows 2000. Its scope does not include Microsoft Windows 95, Microsoft Windows 98, and Microsoft Windows Millennium Edition (Me), except with regard to backward-compatibility issues. The programming languages included in this book's purview are C and C++ when developing Microsoft Win32 and console applications, Microsoft Visual Basic Scripting Edition (VBScript) when working with Dynamic HTML (DHTML), and Microsoft C# when developing software applications within the Microsoft .NET Framework.

Organization of the Book

The material in this book is divided into seven parts:

  • Part I: Introduction
  • Part II: Globalization
  • Part III: Localizability
  • Part IV: Localization
  • Part V: Testing
  • Part IV: Tools and Technologies
  • Part VII: Appendixes

Part I introduces you to the fundamentals of software internationalization and to concepts used throughout the book. Part II, which comprises the bulk of the book, addresses crucial globalization issues. Whether you are planning to localize your application or not, Part II provides essential concepts and technical guidance.

Part III examines how to make your software application and content localizable, whereas Part IV discusses guidelines that are essential for software and content localization and identifies issues that can hinder your product's localization. Part V shows how to conduct testing and contains a handy testing reference: "Sample International Test Cases." Part VI offers a brief sampling of useful tools and technologies that Microsoft has made available for creating a world-ready application, and points you to additional references. Finally, Part VII contains an assortment of other valuable reference material as well as the book's main glossary. (You can find more information about the purpose and content of the book's divisions within each part's introduction.)

In addition to the main glossary, you will find smaller glossaries within each chapter itself. These glossaries are intended as a convenient reference in case you are not familiar with terms that appear within that chapter. The main glossary contains all chapter glossary terms in addition to other terms.

Companion CD-ROM

Accompanying Developing International Software, Second Edition, is a companion CD-ROM, which includes an electronic version of the book; tools and utilities related to globalization; sample applications; and additional resources—code page tables, mapping tables between Unicode code points and code pages available on Windows platforms, HTM Auto Layout (HAL), Microsoft IME Application Interface, and a sample business case for international development.

Contents

The companion CD-ROM contains the following items:

  • Extras folder with a variety of international-related documents that include:
    • Windows and Original Equipment Manufacturer (OEM) codepage tables
    • Mapping tables between Unicode code points and code pages available on Windows platforms
    • An HTML Auto Layout Rules (HAL) document to create HTML pages, dialogs, and user interfaces that need no resizing after translation
    • A sample business case for international development
    • Input Method Editor (IME) documentation that gives a brief overview of Microsoft IME 2002the Japanese version APIin terms of functionality and API format
  • Samples folder, with samples and coding solutions for various international-related applications and demonstration tools. Included are examples involving Microsoft Win32, Web, and Microsoft .NET Framework applications.
  • Tools folder, with some handy, international-specific tools.
  • A fully searchable electronic version of the book (eBook).

Using the CD-ROM

To use the companion CD-ROM, place it in your CD-ROM drive. Autorun will launch a starting menu. If the starting menu does not launch automatically, it can be accessed by running StartCD.exe in the disc's root folder.

About the eBook

The companion CD-ROM includes an electronic version of the book that you can view using Microsoft Internet Explorer 5.01 or later. To use the eBook, select the eBook option from the Autorun menu or run Autorun.exe in the eBook folder and follow the instructions that appear on your screen. If you do not have Internet Explorer 5.01 or later installed, the eBook setup program will offer to install Internet Explorer 5.5 components for you. Setup has been configured to preserve your current settings and associations. If you are using a system running Microsoft Windows NT 4.0 or later (including Microsoft Windows 2000 and Microsoft Windows XP), you will need administrative privileges to install the eBook.

Note Appendixes A-Y of the book appear as PDF files (viewable with Adobe Acrobat) that can be accessed through the CD-ROM menu.

Web Links

All Web links that appear were valid at the time this book was written. Due to the dynamic nature of the Internet, however, it cannot be guaranteed that these links will remain the same. If you find that one of the links is no longer current, try searching for the topic of interest at http://msdn.microsoft.com, http://www.microsoft.com/globaldev, or http://www.microsoft.com/globaldev/dis_v2.

Questions and Feedback

The technology and international conventions included in this handbook are likely to evolve from the time of this writing. Some information in the appendixes, such as address formats and paper sizes, is provided to show you the kinds of defaults that change from country to country. Always double-check this information when applying it to your product. Updates and any corrections to material in this book will be published at http://www.microsoft.com/globaldev/dis_v2. Please send questions and feedback to Dr. International - drintl@microsoft.com.


Part I: Introduction

Part I explores what it means to create and design world-ready software for internationalized products and offers vital planning tips. Chapter 1, "Understanding Internationalization," introduces you to two key processes related to creating internationalized products: world-readiness and localization, while breaking down world-readiness further into globalization and localizability. Other concepts such as locales, locale awareness, and localization are also examined. Finally, Chapter 1 outlines the broader issues you'll need to consider when shipping international products.

In Chapter 2, "Designing a World-Ready Program," you'll learn about the steps inherent to designing software for world-readiness. More specifically, the chapter highlights the issues you need to consider when writing specifications, when customizing features, as well as when designing the user interface and code for a single-binary, world-ready program. Chapter 2 also explores strategies for setting up a project team and a development environment, and gives tips regarding testing and translation issues. In short, Part I provides an overview of fundamental concepts essential to creating internationalized products, with an emphasis on developing world-ready software. These concepts are the backbone for many of the technical details that appear later in the book.


Chapter 1: Understanding Internationalization

Developing internationalized products is a continuous balancing act. Developers and their managers often grossly underestimate the level of effort and attention to detail required to create either a world-ready, single-binary application ready for use in many different markets, or high-quality, foreign-language editions of a product. If you are a developer, make sure your management understands what is involved. Familiarity with the kinds of problems that arise will help you make decisions at the beginning of your product cycle, saving you time and money-not to mention grief.

The process of creating internationalized software has two facets (see Figure 1-1):

  • World-readiness. This facet covers generic coding and design issues and comprises two major areas-globalization and localizability. (See "World-Readiness" later in this chapter for information on globalization and localizability.)
  • Localization. This involves translating and customizing a product for a specific market.

For example, using the National Language Support (NLS) supplied by the Microsoft Win32 application programming interface (API) is a world-readiness step, whereas modifying the user interface (UI) elements, translating text, and standardizing terminology are localization steps. Developers write code, so they tend to focus primarily on world-readiness issues. But because code and feature design affect how a product is translated and customized, developers must also understand basic localization concepts.

Figure 1.1 Software internationalization process.
Figure 1.1 - Software internationalization process.

The first two chapters of this book, which comprise Part I, "Introduction," provide a helpful introduction to software internationalization on Microsoft Windows platforms. They discuss general concepts and techniques for creating world-ready applications, define many of the terms used throughout the rest of the book, and provide a framework for later chapters, which cover specific technical details about using Windows-based tools and Win32 API functions. Part II, "Globalization," covers the many steps needed to globalize a product, from handling many different languages and scripts to displaying data in a way that's appropriate to a specific culture. Part III,"Localizability," explains various localizability techniques developers can use in designing and writing their software to make the localization process easier and more cost-effective for the localizer. Part IV, "Localization," meanwhile, provides a background on the localization process itself.

Part V, "Testing," gives development teams insight into the steps necessary to test world-ready software. It also shows how to test the product's localizability without having to translate it. Part VI, "Tools and Technologies," gives an overview of the many components Microsoft offers designers and developers in the creation of world-ready software. Finally, Part VII, "Appendixes," provides references for many aspects of developing world-ready products. Some examples of these helpful resources include international keyboard layouts, selected international punctuation symbols, Win32 country-specific and language-specific information, as well as international address formats.

As an additional resource, included with this book is a companion CD-ROM containing sample code, Windows code pages, international keyboard layouts, double-byte character set (DBCS)/Unicode mapping tables, along with other ancillary material. The CD also contains an electronic version of this book.


Locales

Glossary
  • Locale: The collection of features of the user's environment that is dependent on language, country/region, and cultural conventions. The locale determines conventions such as sort order; keyboard layout; and date, time, number, and currency formats. In Windows, locales usually provide more information about cultural conventions than about languages.
  • Locale-aware: Exhibiting different behavior or returning different data, depending on the locale. For example, the Win32 sorting functions return different results depending on the locale parameter sent to each function.
  • National standard: A linguistic rule, measurement, educational guideline, or technology-related convention as defined by a government or by the International Organization for Standardization. Examples include character sets, keyboard layouts, and some cultural conventions, such as punctuation.
  • Syllabary: A set of written characters in which each character represents a syllable (for example, a consonant sound followed by a vowel sound). Examples of syllabaries include Japanese katakana and hiragana and the Indic scripts.
  • Script: A collection of characters for displaying written text, all of which have a common characteristic that justifies their consideration as a distinct set. One script can be used for several different languages (for example, Latin script, which covers all of Western Europe). Some written languages require multiple scripts (for example, Japanese, which requires at least three scripts-the hiragana and katakana syllabaries and the kanji ideographs imported from China). This sense of the word "script" has nothing to do with programming scripts such as Perl or Microsoft Visual Basic Scripting Edition.
  • Hiragana: The Japanese cursive script. Each hiragana character represents a phonetic syllable.
  • Katakana: A Japanese script of phonetic syllables, chiefly used to spell words borrowed from other languages. Each katakana character represents a phonetic syllable.
  • Separators: Symbols used to separate items in a list, mark the thousands place in numbers, or represent a decimal point. Different locales follow different conventions for separators.

Geographically speaking, a locale is a place. In software terms, a locale is a set of information associated with a place or a culture. Locale information on Windows includes the name and identifier of the spoken language, the script used to write the language, and cultural conventions. (See Table 1-1.) Locale-aware standards include keyboard layouts, default paper sizes and envelope sizes, character sets or character encoding ranges, text directionality (left-to-right or right-to-left, horizontal or vertical), and input methods. Chapter 4, "Locale and Cultural Awareness," discusses different aspects of locale information in detail.

Table 1-1 Selected default information for several locales.

Locale | English (United States) | French (France) | Japanese | United Arab Emirates (UAE) Arabic
Country/Region | United States | France | Japan | United Arab Emirates
Language | English | French | Japanese | Arabic
Written Script(s) | Latin | Latin | Kana, kanji | Arabic
Direction of Text | Left to right | Left to right | Left to right horizontally or right to left vertically | Right to left
Windows-Defined Code Page | 1252 | 1252 | 932 | 1256
Currency Symbol | $ |  | ¥ |
Long-Date Format | January 27, 2004 | 27 January 2004 | 2004 [image] 1 [image] 27 | ,
Short-Date Format | 1/27/04 | 27/01/2004 | 04/01/27 | 04/01/27
Time Format | 1:00 pm | 13:00 | 13:00 | 1:00
Calendar | Gregorian | Gregorian | Gregorian (Localized) | Gregorian (Localized)
Default Paper Size | U.S. Letter (8-1/2 x 11 in) | A4 (210 x 297 mm) | A4 (210 x 297 mm) | A4 (210 x 297 mm)
Decimal Separator | . | , | . | ,
List Separator | , | ; | , | ;
Thousands Separator | , | Space | , | ,

Windows supports a large set of locales, which it tags with language and sublanguage pairs. The sublanguage generally corresponds to a country/region. One way to think of this is in terms of the phrase "X language as spoken in Y country/region." The way people speak or write a particular language might not necessarily change dramatically from one country/region to the next, but cultural conventions and national standards often differ nonetheless. However, there are other locales where the language can significantly change between country/region, as in the case of Brazilian and Portuguese locales.

Microsoft Windows XP supports 135 locales. Among these are five Chinese-language locales (Hong Kong SAR, Macau SAR, the People's Republic of China, Singapore, and Taiwan), 13 English locales (Australia, Belize, Canada, Caribbean, Ireland, Jamaica, New Zealand, Philippines, South Africa, Trinidad, the United Kingdom, the United States, and Zimbabwe), and six French locales (Belgium, Canada, France, Luxembourg, Monaco, and Switzerland). In addition, Windows XP supports 16 Arabic locales plus nine Indic locales.


World-Readiness

Glossary
  • World-ready: A program that has been properly globalized and developed for ease of localization (known as localizability).
  • Globalization: The process of developing a program core whose features and code design are not solely based on a single language or locale. Instead, their design is developed for the input, display, and output of a defined set of Unicode-supported language scripts and data related to specific locales.
  • Localizability: The design of the software code base and resources such that a program can be localized into different language editions without any changes to the source code.
  • Diacritic: A character that is attached to or overlays a preceding base character. For example, a mark placed over, under, or through a Latin-based character-such as "~"-to indicate a change in phonetic value from the unmarked state. Most diacritics are non-spacing characters that don't increase the width of the base character.
  • Accented character: A character that has a diacritic attached to it-such as "ñ."
  • Code page: An ordered set of characters of a given script in which a numeric index (code-point value) is associated with each character. In this book, this term is generally used in the context of code pages defined by Windows and can also be called a "character set" or "charset."
  • NLS API: Acronym for National Language Support API. The set of system functions in 32-bit Windows containing information that is based on language and cultural conventions.
  • Single binary: A functional binary that is fully globalized and can be used as is for any language version of the software.

"World-ready," "international," "international-aware," "internationalized," "globalized," and "worldwide" are all buzzwords describing programs that are designed to function for more than one language. In order to create well-functioning world-ready software, you will invariably have to rework programs whose code and features focus on a single language, but it is still far more efficient to create a single-binary, world-ready core that can serve as a foundation for all language editions of a product when product development begins than to postpone this effort. This valuable investment involves designing a user interface, a set of features, and a code base that are generic enough to work for most of the product's intended language editions. Of course, some customization might be necessary, but the fewer the changes needed for international editions, the faster the product can be released.

The goal of world-readiness is to present users with a consistent look, feel, and functionality across different language editions of a product. (Consistent terminology is especially important for product suites.) Users expect localized software to support the same basic set of features as the product's native language edition, with the same level of quality. They also expect different language editions to interact smoothly with one another. A single document format that all editions can load and interpret correctly is therefore essential. For example, employees at a Belgian bank might use the French edition of your program to load a document that a colleague created in the Dutch edition of your program. If documents are unreadable from one language edition to the next, or if they convert with numerous errors, your product will get a bad reputation. Understandably, consumers in foreign markets become frustrated when software companies overlook what is for them key functionality. That is why one tries to develop software that can handle any language anywhere.

The team that created Windows strove to create a user interface that would be recognizable in any language. As you can see from the different versions of the Windows XP Control Panel shown in Figures 1-2a and 1-2b, the menus and icons in the English and Japanese editions are consistent. If you don't speak Japanese but are familiar with the English edition, you can probably find your way around the operating system.

Figure 1.2 a Control Panel in the English edition of Windows XP.
Figure 1.2 - a Control Panel in the English edition of Windows XP.
Figure 1.2 b Control Panel in the Japanese edition of Windows XP.
Figure 1.2 - b Control Panel in the Japanese edition of Windows XP.

Just as visual consistency is one measure of how well a product has been made world-ready, so is its support for international conventions. The most basic of these is support for entering and displaying non-Latin characters. In the past, English-language programs commonly limited valid characters to the ASCII character set. This limitation is no longer acceptable, especially with the advent of Unicode. (See Chapter 3, "Unicode.") With the support for multilingual data, it is now possible to create software that can display documents created in almost any language edition of your program.

Other international conventions include rules for sorting strings and for formatting dates, times, numbers, and currencies. The Windows operating system carries a large amount of international information, which you can access through the Win32 NLS API. (See Chapter 4.) This set of functions will help you create world-ready core code that will allow diverse users to enter culturally accurate data.

The part of internationalization that the user doesn't directly see involves coding, testing, debugging, and the mechanics of translation. If you are unfamiliar with the requirements of producing translated software, you will be surprised at the number of details that need to be resolved. This book describes a number of scenarios you might encounter, but it can't possibly predict every situation. The job of a development team is to make sure that whatever needs to be changed for different language editions of a program can be done quickly and easily, without breaking any features. This requires organizing the program's sources intelligently, anticipating how a language edition might be customized in the code design, and avoiding coding practices that cause bugs in translated software. Chapter 2, "Designing a World-Ready Program," and the chapters within Part III describe these techniques in detail.


Localization

Glossary
  • Localization: The process of adapting a program for a local market, which includes translating the user interface, resizing dialog boxes, customizing features (if necessary), and testing results to ensure that the program still works.
  • Levels of localization: The amount of translation and customization necessary to create different language editions. The levels, which are determined by balancing risk and return, range from translating nothing to shipping a completely translated product with customized features.
  • Enabling: Altering program code to handle input, display, and editing of bidirectional or East Asian languages, such as Arabic and Japanese.
  • Layout: The order, positioning, and spacing of text or other user interface elements.

World-ready core code is a basic requirement for software that will ship outside of your native country. You can vary the level of effort and expense devoted to localization, however, depending on the target market and the application type. (See Figure 1-3.) Some markets are currently comparatively small, so shipping a product there is considered an investment for the future. Other markets are well established, and breaking into them requires a large investment and a commitment to high standards. Still other markets are small, but growing rapidly. It is possible to establish a presence in small markets by shipping a partially localized application quickly and following it up later with a more fully localized edition of the same application or its next revision.

At Microsoft, subsidiary marketing managers decide the level of localization for each language edition of Windows. Developers and translators provide the marketing managers with an estimate of how much time and money each level will require. Then the subsidiaries and product teams decide whether the potential return is worth the cost. (For information on specific Microsoft localization choices for Windows, see Appendix V, "Localized Versions of Microsoft Windows.")

Figure 1.3 Levels of localization.
Figure 1.3 - Levels of localization.

The first two levels of localization listed in Figure 1-3, translating nothing and translating documentation and packaging only, involve little development cost-at most, developers will be called upon to explain technical issues to translators. The next level, enabling the code, involves no translation cost. The term "enabling" applies most often to software that was originally designed for European languages. Enabled software allows users to display and enter data in their own languages, even if the user interface isn't localized. Code for Middle Eastern, Indic, or East Asian language editions of such programs must be altered to handle input, layout, display, editing, and printing of text associated with these languages. (If you design and create a single world-ready application, it is already enabled and thus there are no extra costs). Screen savers, multimedia titles, and other programs that do not allow the user to enter or edit text do not require enabling.

If an English-language edition of your product exists, consider using it to test the waters in markets where users will buy a product with an English-language user interface until a fully localized edition is available. The following list gives some guidelines for determining the markets in which an English-language user interface is acceptable:

  • Small markets (for example, Thailand, Philippines,and Indonesia)
  • Markets where your product has no competitors
  • Markets where the target audience speaks English (scientific, medical, and technology communities)

In markets where only translated products are successful, do a partial translation if you want to ship quickly or don't want to pay the cost of a full translation. As a general rule, the more involved the translation, the more questions developers must answer and the more bugs developers must fix.

In more competitive markets, you will need to customize features and, in some cases, add support for hardware that is sold locally.

The Localization Process

Glossary
  • Globalized functionality testing: Functionality testing that has been enhanced to include verifying the world-readiness of a product.
  • Beta testing: Distributing prerelease software to future users and potential customers in order to get feedback and bug reports.

The process of creating localized software involves a great deal of communication between different players on a product team. Figure 1-4 depicts some basic lines of communication and, when read from top to bottom, indicates a rough timeline for international product development. During the core phases, the development team provides files to the localization team, which translates text, resizes dialog boxes, and hands files back for compilation, if necessary. The localized executable then goes to the testing team, which reports functionality problems to development and reports user interface problems to localization. All three groups work together to resolve bugs, and the cycle continues.

The least painful kind of localization process is one in which the product team considers international issues during the initial feature design and coding design stages. Market forces, usability tests, and development constraints also affect the design of the product. Translating and testing the code in parallel with development uncovers problems with code and feature design early in the process, when it is easier to make changes. Localizing a product is akin to launching a probe to Mars. It's much less involved and much less expensive to adjust the course soon after liftoff. The trajectory might be only a few degrees off when the probe is near Earth, but if you wait until late in the mission to make adjustments, you won't have enough fuel to keep the probe from shooting past Mars by a million kilometers.

Figure 1.4 The localization process.
Figure 1.4 - The localization process.

It is important to "freeze" as much of the user interface and feature design as possible well before the product goes into final testing. Freezing the design means that no more changes will be made before the product ships. Every time a piece of the user interface changes, it has to be retranslated. In addition, the teams in charge of documentation, online Help, online tutorials, and marketing collateral might need to update their materials, which also have to be retranslated. Late changes, as you can imagine, cause significant, expensive delays in the release of localized products.

Shipment of International Products

Glossary
  • Release delta: The time between the release of the domestic product and the release of the localized edition.
  • Simultaneous ship, or "sim ship": The release of localized editions of a product at the same time as the domestic product; or a short release delta, usually within 30 days (to allow for nondevelopment needs).

You can take one of two approaches to shipping international products. The first is to begin working on international editions after the domestic edition has been released or when it is almost finished. The second is to plan for international products in advance, work on several language editions concurrently, and ship them all at roughly the same time.

Those who would advocate the first method fail to realize how much effort it takes to create localized software. If it were an easy task, the text of this book would be only a few pages long. A common argument from development managers is, "Our domestic product is really important. We don't want to delay the domestic product or distract the team by worrying about the international editions now." This short-term, restricted thinking makes sense only for companies that generate 95 percent of their revenue domestically or that have no international competitors. In other words, very few companies benefit from this strategy.

Microsoft's careful attention to its internationalization practices is the reason it earns the majority of its revenue outside the United States. Companies that make most of their revenue domestically either ship products with a domestic focus, such as financial software or certain multimedia titles, or they haven't tapped the potential of the international market. Because controlling costs has a significant impact on profits, the main argument against delaying work on international editions of a program is that to delay such work is expensive.

One effective approach, and the one Microsoft uses to develop Windows, is to create a solid world-ready core code base and to begin translation work as soon as there is something to translate. Microsoft develops the English, German, and Japanese editions of its operating systems in-house and in parallel-English because it is the largest language market for Windows and because most of the developers on the Windows team are native English speakers; German because as the largest European-language market, it is a good test of European-language functionality and ease of translation; and Japanese because East Asia is Microsoft's second-largest market, with Japanese being a good representative of East Asian language-development challenges. Other language editions, such as Italian and Swedish, are translated in parallel at Microsoft's subsidiary in Ireland. Teams in both Ireland and Redmond, Washington, worked furiously to ensure that localized editions of Microsoft Windows 2000 and Windows XP were ready for testing at each major development milestone.

This method of developing international software is easier to implement if all developers are held accountable for the international functionality and localizability of their own features. Naturally, having an internationally conscious team that constructs several language editions of a product concurrently-and with minimal problems-is an ideal that might take more than one product cycle to fully achieve. Again, the key is to begin early in each new product cycle. Early translation, for example, can uncover design blunders before complex features have been built around them. Coding problems can then be resolved while the code is still fresh in developers' minds.

The best result of a "three-pronged" development approach-such as Microsoft's simultaneous development of English, German, and Japanese editions-is simultaneous shipment of more than one language edition. The Windows 2000 team, for example, released the French, German, Spanish, and English editions of its product within days of one another. A "four-pronged" approach, such as the one summarized in Table 1-2 on the following page, would include either a right-to-left language (such as Arabic), a European language that does not use Latin script (such as Russian), or a language that is based on a complex script (such as an Indic language).

The international press pays attention whenever a new product revision reaches the market. Having more than one language edition ready when the domestic edition is released allows more than one language edition to take advantage of the publicity surrounding the new revision. Not having key language editions ready soon after the domestic release means that international customers will wait for the revised localized releases rather than buy the existing localized releases. The longer they have to wait, the more frustrated they become, and the more sales your company loses.

You will not significantly delay your domestic product if you work toward international functionality from the beginning of your product cycle; you will actually save your company time and money in the long run. Inventing a product-wide plan that satisfies all internationalization requirements-timely domestic and international releases, inexpensive localization, a compelling set of new features, and appropriate staffing-can be a balancing act. But without a plan and your team's commitment to it, producing international editions of your product will turn into one headache after another.

Table 1-2 Sample "four-pronged" approach, designed to uncover potential problems in international software early in the development process.

Language | Purpose | Reason | Language Alternatives
English | To develop product and test for general functionality | Largest market for Windows-based software | Native language of developers
German | To uncover international- related bugs and over- crowded user interface designs | Largest European language market | Finnish, French, Spanish
Japanese | To enable large non- Latin character sets, vertical writing, and vertical printing | Largest East Asian language market | Simplified Chinese, Traditional Chinese, Korean
Arabic | To create right-to-left look and feel; enable bidirectional text | Largest Middle East- ern market; issues are a superset of Hebrew issues | Farsi, Hebrew

If your product team has already produced localized editions, but they have long release deltas, the principles in this book can help you shorten the release deltas for your next product revision. In some cases, a new product plan might schedule localized releases of Product Two within months of localized editions of Product One. If this happens, suspend localization of Product One and concentrate on shipping international editions of Product Two quickly. Microsoft, for example, did not release a Japanese edition of Microsoft Visual Basic 3, but went straight from localizing Visual Basic 2 to localizing the version of Visual Basic that followed Visual Basic 3.


Summary

Developing world-ready software for an internationalized product involves creating a functional core code base and a generic feature design that can serve as the basis for all language/locale editions of a product. This generic feature design (globalization) needs to take into account items like language; written script; sort order; input method; paper sizes; as well as date, time, number, and currency format that changes from locale to locale. The level of localization required for a program to be successful depends on the target market. The more effort you spend, the higher the risk and cost, but the greater the potential return. A major factor in keeping the localization costs down is to make sure the software and all of its supporting content (documentation, Help files, marketing material, and so on) are easily localizable (localizability). The most effective process for creating localized products takes world-readiness considerations into account at the beginning of the product cycle and throughout development, and expedites the shipping of multiple language editions.


Chapter 2: Designing a World-Ready Program

People who plan ideal cities spend months and sometimes years researching, documenting, and revising their proposals. It doesn't make sense to spend years planning most commercial software projects, but the complex process of planning a community illustrates some important principles. City planners study other cities and talk to the people who live in them to determine how they should design a new community. They do this before they even dig the first hole to lay a foundation for a building or a street. After all, a city has to attract residents, or building it will be a big waste of money. And business districts and neighborhoods must have room to grow. Any city budget director will tell you that having to tear down an existing high-rise or apartment complex to make room for a hospital, library, or police station is expensive.

A decision to release your program in different language editions-or at least to have it handle many different scripts (Latin and other scripts)-affects decisions that your team makes regarding coding and user interface (UI) design during the product cycle. A good underlying program design will work for all language editions of a product. Even if you have never designed or created world-ready software, you can probably foresee some of the issues that might arise if you examine an existing program. As with any project, large or small, the golden rule for creating world-ready software is to plan ahead. Few tasks in software development are more arduous, time-consuming, and expensive than rewriting existing code and redesigning existing features so that they will work for other languages. With a good plan, you can create a single-binary application that will accommodate multiple languages instead of just one. Some advantages of creating a single, world-ready binary include the following:

  • Shipping one functional core binary to all platforms and for all different language versions significantly reduces your development hassle and costs.
  • Conditional compiling becomes unnecessary.
  • There is no need to maintain separate source codes and development teams.
  • All your language versions can be shipped almost simultaneously, which means your sales force can capture more revenue from these versions.
  • Software updates are simpler because the same version of patches can be applied to all languages without any additional engineering effort.
  • Customers' global Information Technology (IT) departments save time and money because they only need to deploy a single service pack instead of many different versions.

Only a few years ago, creating a single, world-ready binary was considered a costly and almost impossible task to achieve. But with the unprecedented international support that Microsoft Windows 2000 and Microsoft Windows XP offer, the process has become much easier and more cost-efficient. As a result, with a little extra effort, you can reach a much larger market than was possible earlier, with fewer competitors. In addition, since most corporations are multinational, they would rather distribute and support one set of software executables for their worldwide operations than be forced to maintain a mishmash of binaries.

Indeed, the Windows XP team has learned from experience to use the single-binary approach. For example, the Chinese, Japanese, and Korean editions of Microsoft Windows NT 4 had slightly different application programming interfaces (APIs) for Input Method Editors (IMEs), the modules that handle the input of ideographic characters. (See Chapter 5, "Text Input, Output, and Display.") The differences made it hard for developers to create single executables that would work on all East Asian language editions of Windows NT 4. Windows XP, in contrast, has a single unified IME API for not only the East Asian editions but for all the language versions of the operating system. Developers who wrote applications for East Asian and other non-Latin versions of Microsoft Windows prior to the release of Windows 2000 (the first release of a single binary for Windows) were no doubt pleased with this improvement.

This chapter describes the issues you need to consider when writing specifications, when customizing features, as well as when designing the user interface and code for a single-binary, world-ready program. It also describes strategies for setting up a project team and a development environment, and gives tips regarding testing and translation issues. By familiarizing yourself with the concepts presented in this chapter before you begin your next software project, you will be on the right track towards creating a successful world-ready application.


Identifying World-Ready Requirements in Specifications

The design process will generally begin with your specification, in which you identify all criteria that will help bring about world-readiness. The general categories to consider when writing your specification include globalization, customizing features, designing a user interface (UI) that's localizable, legal issues, and accessibility of features. Each of these areas is discussed in more detail throughout the remaining sections of this chapter.

More specifically, your team's specification should address factors affecting your ship dates, such as staffing requirements and the scheduling of milestones for coding, translation, and testing. In addition, the members of your team who are writing the program specification need to know and specify in advance which languages and locales the final product is targeting and to what extent each language edition will actually be translated.

Also, when describing the appearance and functionality of new features, the team should be sure to address internationalization issues in the specification. Some examples include whether certain features will be removed or customized for different editions, and whether a feature's behavior must be locale-aware. For instance, what happens if a user enters Cyrillic (Russian) characters? Is it necessary to adjust certain dialog boxes for the Japanese edition? Program designers need to consider such details. If they are not familiar with foreign languages or with the needs of international users, they need to talk to people who are.

If you work for a large company, send copies of your draft specification to marketing representatives and language specialists at international subsidiaries. If you work for a small company, share the specification with overseas partners or consultants. Your partners should be able to point out any glaring errors or oversights and give you feedback on the relevance of proposed new features and designs to their markets. They might forward customer reports, market-research results, and reviews of competitive products from the local press. They should also provide you with locale-specific information your product might need, such as typical postal codes, business letter formats, punctuation rules, and so on. (Country-specific formats for addresses, currency, date, time, and other information can be found in the appendixes of this book and on the accompanying CD.) Familiarity with the specification will in turn help your international partners make plans for marketing the product once it is released. Good planning will help you ship your product quickly. Early feedback on the specification from your team, key customers, usability tests, and market experts will help you determine the best way to implement your features at a much lower cost than the cost of making changes at the last minute. Large-scale, last-minute changes in your product design will cause a chain-reaction delay in the ship date of your executable, Help files, tutorials, documentation, and marketing collateral-all of which might then need to be retranslated and retested. You never want to ship an inadequate product, but major last-minute changes are expensive; you can avoid them by working from a solid, world-ready specification.

Some of the areas just mentioned that you'll need to consider when developing a world-ready specification-for instance, specifying which languages and locales the final product is targeting-overlap with tasks associated with globalization. This makes sense since focusing on globalization issues, once again, is an integral part of writing any specification that's world-ready, as discussed in the next section.


Planning for Globalization

Glossary
  • Encoding: A method or system of assigning numeric values to characters (for example, ASCII, Unicode, Windows 1252).
  • Ideographic character: A character of Chinese origin representing a word or a syllable that is generally used in more than one Asian language. Sometimes referred to as a "Chinese character."

Although Part II, "Globalization," discusses globalization in detail, it's important to have an initial understanding of what globalization covers as you prepare your specification. Making sure that all language versions of your software have the same built-in capability to support all targeted languages and locales is called "globalization." It is essentially a process that involves the following general tasks:

  • Identifying the languages and locales that must be supported.
  • Designing features that support targeted markets, languages, and locales.
  • Writing code that functions equally well in any of the supported locales.

These tasks are all centered on the concept of locale-awareness, which, as you've seen, should be an integral part of your specification. (See "Identifying World-Ready Requirements in Specifications" earlier in this chapter.) Nevertheless, the scope of globalization features is even broader than issues regarding locale-awareness. A second area of globalization involves encoding data for internal processing and sharing it with other applications. Like all aspects of coding, there are many ways to do this, each having strengths and weaknesses. However, Chapter 3, "Unicode," explains why Unicode is the best encoding method to use when dealing with world-ready applications.

A third major area of globalization concerns the input, output, and display of data to the user. You will recall that IMEs refer to the modules that handle the input of ideographic characters. In order to ensure correct input of data, IMEs are used for languages such as Chinese, Japanese, and Korean. Character-shaping engines handle the display of complex scripts such as those in Arabic, Thai, and Indic languages, among others. Since the input, output, and display of data are all processes linked closely with font technologies, you should carefully consider the best approach for using fonts.

You should also determine how to manage Multilingual User Interface (MUI) technology, the fourth major area of globalization. For example, how will you display items like menus, dialog boxes, and system messages in the user's preferred language? In order to help you better understand the four globalization issues highlighted here, Part II, "Globalization," is exclusively dedicated to this topic.


Customizing Features

Glossary
  • Dynamic-link library (DLL): A module containing functions or resources that other programs or DLLs can utilize. DLLs cannot run by themselves; other programs have to load them.
  • Kanji: The Japanese name for ideographic characters of Chinese origin.

The process of writing a specification that's oriented toward world-readiness will, among other things, force you to think about what features will need customizing. Of course, a well-designed, world-ready core product minimizes the need for customizing features for different language editions. As you write your product specification, however, you might find that certain features important to your domestic market don't make sense for other markets, or that some markets require functionality that isn't necessary for your domestic market. For example, the East Asian versions of Windows XP automatically add the necessary IMEs at installation, whereas the English version does not, but instead allows the user to install them later if needed.

Some features might be essential for all language editions of a program, but they might require that different code be executed for different languages. One obvious example involves features that do lexical analysis-spelling checkers and grammar checkers, for example. Spelling rules and common words change from language to language. Therefore, most Windows-based software products ship spelling engines as separate DLLs and spelling dictionaries as separate compressed files. For some languages, such as Italian, spelling rules are simple and consistent. For others, such as Japanese, spelling checkers and grammar checkers are exceedingly difficult to write.

Another example of feature design that is locale-specific involves the Regional And Language Options component of Control Panel in Windows XP. While the Sorting tab appears in the Customize Regional Options property sheet of some language versions, in others it is absent, depending on the particular language's rules regarding word sorting. For instance, some regions like English (United States) or Italian (Italy) have only one way to sort words in some prearranged order. Other regions like German (Germany), Spanish (Spain), or Simplified Chinese (PRC) have more than one way. Thus the Customize Regional Options property sheet displays the Sorting tab only when the region's locale has more than one sort order, such as in the German locale, which has two different sort orders (Dictionary and Phone Book). (See Figure 2-1.) Since Italian has only one way to sort, the property sheet for the Italian locale does not include the Sorting tab.

Figure 2.1 The Customize Regional Options property sheet for the German (Germany) and Italian (Italy) locales.
Figure 2.1 - The Customize Regional Options property sheet for the German (Germany) and Italian (Italy) locales.

In cases such as those just discussed, it is important to create code that is flexible, so that you can easily add, remove, or customize functionality. This is especially true since your product will face tough competition in well-established markets. In Germany, for example, word processing programs with poor hyphenation engines tend to sell poorly, and in Japan, people will buy only the text processing programs that make it easy for them to enter kanji characters. Arabic speakers might expect the ability to juxtapose Arabic and English text.

Features that access large amounts of language-specific data can use text files to simplify customization. Windows Setup, for example, reads from text files the list of shipped components along with their localized file names, file sizes, and floppy disk locations. Localizing the Setup program for Windows, then, is only a matter of translating the program's user interface and tailoring the text files. The core setup code works for all language editions. Many spelling checkers use a similar mechanism. The code that searches for spelling errors and suggests corrections remains constant, but special dictionaries-such as medical, legal, and scientific dictionaries-are provided in separate data files.

Designing an International User Interface That's Localizable

Glossary
  • Hard-coding: Putting string or character literals in the main body of code, such as the .C files or the .H files, instead of in external resource files. Basing numeric constants on the assumed length of a string or having any assumptions about language- or culture-specific matters fixed in the code (such as length of strings, formats of dates, and so on).

By now you have seen that creating world-ready applications produces versatile applications that can be used in many different locales, thus giving you access to more markets and increased revenue. Another reason for creating world-ready applications is to keep localization and development costs down, with localizability of the UI being one of the best ways to accomplish this goal. Though both globalization and localizability are part of world-readiness (which, along with localization, forms the basis of creating internationalized software), it's important to understand the distinction between the two.

Take, for example, the executable Wordpad.exe. Making sure users can edit any supported script using different keyboard layouts and IMEs-or displaying calendar information that's specific to a particular language when users insert the date in WordPad-is part of globalization. The product does not need to be translated into a given language for users to leverage these functionalities and for testers to test these features.

Now, suppose you want to translate WordPad into a given language. Making sure that all resources that need to be translated are separated from your source code, and that dialog boxes are designed in a way that can accommodate text expansion, are both localizability considerations. Since globalization deals mostly with the application's functionality and not its user interface, defects pertaining to it are easy to detect early in the design process before any translation even starts. However, these defects require you to address potentially major changes in design or extensive coding changes. Localizability issues, on the other hand, are harder to detect. The software first needs to be either code-reviewed or be translated into either a pilot language or a pseudo-language. (See Chapter 12, "Testing Localizability with Pseudo-Localization.") Also, sometimes issues only affect a given language, so you will need some extensive language-specific testing to unveil certain problems or bugs. On a positive note, localizability defects are usually relatively easy to fix, assuming that it's not too late into the design process to make such changes. Things can backfire, however, if you wait too long to address bugs, since you will then need to correct them in every language version you ship.

When designing for localizability, remember that it is human nature to take for granted that which is comfortable and familiar to us. For example, linguistically speaking, many grammar rules of our native language, which we learned intuitively when we were young, tend to reside in our unconscious mind. We usually activate or use these rules also somewhat unconsciously, rather than with our conscious, analytical mind. Similarly, it can be tempting or easy to assume that, for example, the idiomatic expression that something "costs an arm and a leg" will be similar in other languages. However, in Spanish, the same expression is that something "cuesta un ojo de la cara," or literally, that something "costs an eye from the face." In similar fashion, with languages in applications, it can be easy for developers to lose sight of how these languages will function in another language market, unless they consciously analyze the issues beforehand. Many developers create coding tricks that they feel are clever in the earnest attempt to save work and money, yet which often overlook the broader context.

Some of these programmers' tricks include the following:

  • Hard-coding strings in their code, to avoid having to create a resource handling service.
  • Saving storage space by using text strings that are concatenations of phrases. For example, developers might use the text string "The [flood gate/control rods/exit door] cannot be [closed/open/reset]"-which doesn't account for changes in verb form resulting from nouns of differing gender and number-instead of having to create nine different sentences.
  • Overlapping controls of the dialog box (such as hiding one button behind another), thus saving display space in the dialog box.

Although these strategies can be viewed as clever in that they can save some work for the developer, in the long run these myopic practices will increase costs when you localize your applications. This is because every code change that could have been avoided with good localizability practices is multiplied by the number of language markets for which you are preparing your applications. For example, if you're going into 24 markets, the cost of every localizability issue not addressed in the core code costs you up to 24 times its original cost to fix, because you might have to fix it in every language version. Chapter 7, "Software Localizability Guidelines," discusses the many areas you need to think about when creating a localizable user interface.

Researching Legal Issues

Another component of customizing for international markets entails investigating pertinent legal issues. As you get feedback on your product specification, solicit advice from people familiar with the law in local markets. Certain kinds of features might affect the international distribution of your product. For example, some countries have export and import laws that govern the sale of products that contain encryption algorithms, such as those used for file compression or copy protection. Software that connects with telephones can be more difficult to sell in Europe, where some governments closely regulate the use of phone lines. Software and documentation that is marketed in France must be in French, and software marketed in Brazil must support hardware that is developed locally.

In Germany, competition law severely restricts a company's ability to claim that its product is better than another company's product. This directly affects advertising and packaging, but it can also affect sample files, documentation, Help files, or features that assist users in transitioning from a competitor's product to your product. Productivity features, such as those that keep track of how long a user has been editing a document, might conflict with labor laws.

Another important legal consideration is, of course, software piracy. Microsoft works with the Business Software Alliance (BSA), located in Washington, DC, to protect its products overseas. (Figure 2-2 on the following page shows one of Microsoft's logos used in literature to stop piracy.) The BSA conducts educational campaigns and lobbies governments for tougher piracy laws. The efforts of the BSA have proven more effective in preventing piracy than adding copy-protection code or hardware devices to products. Although counterfeiting is illegal in many countries, some countries still allow local companies to market clones of software products. Before you enter a new software market, research these issues. Keep in mind that entering new markets requires diplomacy. Microsoft's well-publicized efforts to establish Windows in mainland China is a good case study for any company seeking to expand its operations. Microsoft was not aware of the politically sensitive issues of developing products offshore for import into the People's Republic of China when it began developing and marketing the Simplified Chinese edition of Windows. As a result, the company unwittingly angered the Chinese government. For this reason, Microsoft worked very closely with the Chinese government and local developers to ensure the full success of Windows XP.

Figure 2.2 One of Microsoft's anti-piracy logos.
Figure 2.2 - One of Microsoft's anti-piracy logos.

Keeping Features Accessible

Glossary
  • Accessibility: The extent to which computers are easy to use and available to a wide range of users, including people with disabilities.

The purpose of making your software world-ready is to make it easy for as many people around the world as possible to use it. Designing for accessibility has the same purpose-and many features that place software within reach of people with visual, aural, cognitive, or mobility impairments also help prepare it for localization. For example, dialog boxes with a simple design and minimal text are easier to translate and easier for some people with visual impairments to read. Addressing accessibility issues in your product specification will result in a product with a larger potential worldwide market. Just as with world-readiness, designing your product to be accessible from the start saves you from having to redesign it later-and possibly from having to release special editions.

Detailed guidelines for designing accessible applications for Microsoft Windows are published in the subtopic titled "Accessibility" in "User Interface Design and Development," available in the Microsoft Developer Network Library.


Organizing a Product Team

Identifying world-ready requirements in your specifications and customizing features for the global market are two essential steps toward designing a world-ready program. Of course, having people who will implement your plans is obviously another vital component. Putting together an efficient product team is a challenge that can make a significant difference to your bottom line. If your entire team-including developers, designers, testers, translators, marketers, and managers-is committed to all language editions of your product, and if management holds them responsible for all language editions, you won't have to hire a lot of extra people.

The golden rules for product team managers are as follows:

  • Educate all product team members on internationalization issues.
  • Hold all team members responsible for the international functionality of their features.
  • Develop at least one localized or pseudo-localized edition in parallel with the domestic edition. (See Chapter 12.)

Managers might argue that holding all team members accountable for all international products will cause them to lose focus, but in reality such a policy will save time and effort in the long run. Developers who are responsible for only the domestic edition of a product have no incentive to design global code, and might not even know how to do it. Later, when problems are uncovered during software translation and testing, those developers will have to spend time fixing errors that would have been easy to avoid; instead of working on the next version of the product, they will be busy redesigning old code. If your team creates a program so domestically centered that bitmaps must be changed or features redesigned, you will sacrifice consistency among language editions. If you choose to minimize the time spent fixing internationalization problems, you will release poor-quality products. Fixing and adjusting an existing product takes time, during which your company pays salaries and loses sales.

Microsoft used to have separate teams working on each product-one for domestic software and one or more for international software-and each one reported to separate managers and had different objectives. The domestic teams wanted to finish their English products as quickly as possible, with a minimum of interference. The international teams wanted to release localized editions as quickly as possible, and constantly needed domestic teams to make changes that would lower localization costs. This arrangement created an "us vs. them" mentality, because each team invariably had different priorities and felt the other was making its job harder.

Eventually Microsoft merged the domestic and international teams, but managers still assigned responsibility for international editions to "international developers." Just as before, these developers had to contend with algorithms that were not designed for multiple languages, and they had to decipher code that other people wrote.

This old approach, with its sense of being disconnected, contrasts with the current approach, with its emphasis on being a cohesive unit. The results are clear. For the Windows XP project, the German language edition of the system was shipped the same day as the English edition. In addition, many language editions-including Chinese, Japanese, Korean, and several European languages-shipped within 21 days of the English release. This was an improvement of more than a year over the release delta between English and Japanese Windows 3.1. These goals were achieved because, after Windows 3.1, Microsoft moved all the development for Windows to Microsoft's headquarters in Redmond, Washington. The teams working on features related to East Asian languages, complex scripts, and multilingual versions worked alongside the rest of the Windows XP team and reported to the same managers. The Windows XP team was thus able to meet strict milestones for delivering localized editions of their operating systems.

Although some teams are still tempted to follow the old approach, it has proven to be a very inefficient way to create international products. Developers find it much easier and more rewarding to be the experts on a subset of features than to be jacks-of-all-trades trying to maintain a working knowledge of an entire product code base. And if you hire one or two people to monitor all international functionality, you are basically paying some developers to clean up after other developers who will continue to make the same mistakes. Why pay someone to fix design problems and bugs that educated developers can prevent in the software's planning stages?

As you delegate more responsibility for world-readiness to each member of your development team, remember that changes aren't likely to occur immediately. It takes time to instill global thinking into your company culture. A good example of the need for perseverance can be seen in the development of Microsoft Word. In the earlier versions, there were many different localized code bases. However, persistence has paid off. With team members sharing the responsibility for world-readiness, all localized versions of Word XP are now created from one code base. (See Figure 2-3.)

Figure 2.3 The gradual convergence of localized versions of Word into a single, world-ready binary.
Figure 2.3 - The gradual convergence of localized versions of Word into a single, world-ready binary. 1

In addition to promoting a company culture of world-readiness that trickles down to each individual, you will also need to consider the needs of some key people-your translators. (For more information, see "Making Life Easier for Translators" later in this chapter.) Quick release of localized software requires that translators begin work early. If you plan to hire translators from abroad, remember that it takes time to obtain work visas for noncitizens. Microsoft employed in-house translators and third-party contractors to translate German, East Asian, Arabic, and Hebrew editions of Windows XP in Redmond. All other European editions of the operating system were translated, tested, and manufactured at Microsoft's subsidiary in Ireland. The subsidiary also hired third-party contractors in Europe. If you do hire a third-party localization company, make sure that translators have direct access to members of your product team, or you will lose some of the efficiency and other benefits gained with early translation.

Much of the expense and lost productivity resulting from any inefficient approaches described in this section drain a project's resources subtly over time, and might be apparent only to people who prepare budgets and earnings reports. If you are having a difficult time convincing your company to change its approach to internationalizing software, arm yourself with schedules, bug lists, and sales statistics from past products. Back up that ammunition with market research and testimonials from product team members. Finally, show people this book.


Setting Up a World-Ready Development Environment

Along with assembling your product team, you will need to provide a development environment that is conducive to world-readiness. A key prerequisite to creating a world-ready core code base for your program is that all language editions share the same core source files. Maintaining separate source files for different language editions of the same product is error-prone, a waste of time and disk space, and unnecessary for code that has been made world-ready.

When you have shipped or are close to shipping your domestic product, you might decide to make a copy of your current source files and set up a separate development project for the next product revision. In such a case, you will need to pass bug fixes to the new project and incorporate updated code or features into the old project for international editions that have not yet shipped. When the schedules for your international releases overlap with the schedule for the next product revision, you might be torn between your desire to take advantage of improvements and your desire to ship to international markets on schedule. Obviously, you have to use your best judgment in these situations.

Just as all language editions should share a single set of source files, all language editions should share a single bug database or bug-tracking system. Many defects will be common to all language editions, but bugs that first appear only in non-native editions can reveal design flaws with regard to world-readiness. Those responsible for testing certain features in native editions should also be responsible for testing the same features in the international editions, or with non-native settings and parameters. It's a good idea to have dedicated test scripts for localized products and help from native speakers to test certain features, but the most successful philosophy is that of one product, one team.

As you start a new project, be sure to select tools-such as editors and compilers-that will help rather than hinder world-readiness. If you plan to write customized build tools or design your own resource-handling mechanisms (instead of using Microsoft Win32 resource files), first consider what effect they will have on your international editions. If you must write your own build tools, make them world-ready the way you would your main executable.

Eliminating Compile Dependencies

Glossary
  • No-compile: A process of localization where the code is not recompiled.

A major incentive for creating world-ready core code is not having to recompile source files to create localized editions of your program. If you eliminate compile dependencies by avoiding hard-coded strings or constants and by not putting localizable resources in header files, your localization process will go much faster. (See "Designing an International User Interface That's Localizable" earlier in this chapter.)

Very large programs take hours to compile, even with powerful processors. Building debug and nondebug executables from scratch for each language takes significantly longer (unless you buy a number of expensive machines) and requires large amounts of disk space. If you share common code, you have to compile your main executable only once. To create localized editions, you first separate items like strings and graphics from your source code, and then you compile only these localized resource files and link them to the executable or to a separate DLL. (See Chapter 7 for more details on resource separation. Also, see the chapters within Part III,"Localizability," for other localizability issues in general.) It's even possible to eliminate compilation entirely by using localization tools that allow translators to edit executables directly.

To handle those parts of the code that will change from one language edition to another, you can create a directory structure that separates your core code from each language version of your localized resources. Each set of resources can be placed in a single directory. Translators need access only to the directory containing the native-language files and the localized files for which they are responsible. A multilingual build process can incorporate resource files from the appropriate language directory and place resulting files in that particular language's Build directory. (See Figure 2-4.)

Figure 2.4 Sample directory structure.
Figure 2.4 - Sample directory structure.

All files needing language customization are located in the Resources directory. Developers update files in the Native directory and use batch files to propagate the changes to other language directories. (The Neutral directory separates all your resources that do not need translating from those that do.)

This "no-compile" strategy has a number of obvious benefits. Separating code from localizable resources helps you write world-ready code. Also, significantly reduced compile times allow for faster turnaround in the development, testing, and translation cycle. Testers, for example, have to test only one executable. Translators, especially third-party consultants, don't need to have access to source code-with only an executable, resource files, and a build script, they can compile a localized program to check their work.

Without this "no-compile" strategy, you might need more than one core executable-one for all single-byte, left-to-right languages; one for multibyte languages; one for single-byte, bidirectional languages and several for languages based on complex scripts. Multibyte, right-to-left, and complex-script languages require special input and layout handling. A single code base for Windows XP was used to produce more than 24 different language editions, all of which use a no-compile approach.

Even though source files for Windows contain #ifdef statements to distinguish code bases, #ifdefs are never used to handle cases specific to individual languages. The best place to put #ifdef statements is in header files. Don't use code with #ifdef statements to fix bugs that occur only in the nondomestic language editions, since this practice makes international concerns too easy to ignore. With the increasing importance of multilingual documents, it's harder to label bugs as being "international only." Often the code surrounding #ifdefs is poorly designed and the #ifdef is simply a hack. You can be sure that developers who use #ifdefs and hacks to fix international-related problems will forget to update the code inside the #ifdefs when they update the surrounding code.

You can easily replace language-specific #ifdef statements with run-time if and switch statements. The following fragment from a hypothetical program's startup code uses #ifdefs to omit features for certain language editions:

#ifdef USA
#define DEFAULT_PAPER_SIZE 1 // USLetter
#endif

#ifdef JAPAN
#define NO_SPELL_CHECKER
#define EASTASIA
#define DEFAULT_PAPER_SIZE 2 // A4
#endif

#ifdef KOREA
#define NO_SPELL_CHECKER
#define EASTASIA
#define DEFAULT_PAPER_SIZE 2 // A4
#endif

...

#ifndef NO_SPELL_CHECKER
InitSpellChecker(); // no spell-checker for Japan or Korea

#endif

...

#ifdef JAPAN
InitImperialCalendar(); // This is for Japan only.
#endif

...

#ifdef EASTASIA
InitIME(); // needed for all East Asian builds
#endif

This code is rewritten as follows to use a global variable that is initialized at startup time. Before the program calls any code related to spelling checkers, IMEs, or paper sizes, it checks this variable. (See Chapter 4, "Locale and Cultural Awareness," for a more detailed explanation of PRIMARYLANGID and LANGIDFROMLOCALE.)

typedef struct _LOCINFO
{
int fSpellChecker;
int fUsesIME;
int DefaultPaperSize;
LCID lcidDefaultLocale;
} LOCINFO;

...

LOCINFO locinfo;

GetLocInfo(&locinfo); // function implemented in language DLL

if (locinfo.fSpellChecker)
InitSpellChecker();

if (locinfo.fUsesIME)
InitIME();

if (PRIMARYLANGID(LANGIDFROMLOCALEID(locinfo.lcidDefaultLocale)) ==
   LANG_JAPAN)
{
InitImperialCalendar();
}

...


Making Life Easier for Translators

Glossary
  • Localization kit: A subset of tools, source files, and binary files that can be used to create a localized edition of a program. Generally given to translators or third-party contractors.

Setting up a proper development environment, as just discussed, can go a long way toward ensuring world-readiness. Developers can have a significant impact on the translator's job, a particularly important task since translation is not cheap-most translators charge by the word. Developers can help maximize the accuracy of translation and minimize the amount of support translators need in a number of ways.

First of all, clarify the context of strings that should be translated, preferably by providing comments in resource files or separate text files. Does the English word "integral" refer to the mathematical operator, or is it an adjective meaning "important"? Does a particular string appear on a dialog box, menu, or button, or in a message? If a string should not be translated, try to put it in a separate resource file where the translator won't even see it, or use a standard way of naming identifiers. For example, you could prefix identifiers for debug messages that shouldn't be translated with "IDS_DBG" or with an underline, as in "_IDS."

When composing messages or other translatable strings, watch out for slang, sloppy grammar, and ambiguous words or statements. Translators will have a hard time understanding poorly worded messages, and so will international users. Have a linguist or a professional writer edit messages for clarity before you send them for final translation. This will increase the accuracy of translation as well as improve the quality of your native-language product. If translators know which strings to translate and how to translate them, they will be able to work much faster. Their work will be more difficult if terminology is inconsistent within a single application or among different applications, and if adequate tools for editing and updating resources are not available.

Indeed, there are many localization tools that can expedite translation. Some, for example, allow both you and the translator to create translation glossaries that you can share with other software projects and use for automatically translating resource files. (Microsoft supplies translation glossaries at http://www.microsoft.com/globaldev .) Machine translation software is also becoming commercially available. Automatically translated text still requires editing, but editing often takes less time than translating from scratch.

Giving translators a localization kit will allow them to work more independently because they can compile and check their work as they do it. Passing files back and forth among development, translation, and testing departments is simple when all parties are on the same computer network, but is more difficult when the translators are third-party contractors who work in different countries. Express-mailing floppies, CD-ROMs, or portable hard drives is one option, as is exchanging files via the Internet.


Testing Considerations

Just as localization should begin early in the product cycle, so should testing of localized programs. It's very tempting to concentrate all testing efforts on the domestic product in order to get it shipped quickly. Early testing of all editions, however, ensures that development is paying attention to international issues because it often uncovers serious bugs in the core code.

When deciding whether to fix a bug, determine how much time fixing it will require and how crucial the fix is to the finished product. Does the bug make the program unusable? If not, will it seriously affect sales in any important target markets? Maintain common standards for all language editions of your product. Never postpone fixing bugs found in your domestic software product that will affect international users.

At the end of the product cycle, when the domestic product is close to shipping, it is tempting to postpone fixing bugs that arise only in localized software until after the domestic product ships, out of fear of "destabilizing" the domestic edition. But every time you delay fixing an international-related bug, you are postponing all of your international releases. If your domestic product will ship in a few days, postponing a few bugs exclusive to localized editions is not a big deal. However, many experienced developers have worked on projects that were in ship mode for weeks, months, or even years. Under these circumstances, allowing international bugs to pile up is dangerous and leaves unresolved a significant number of changes that you will have to merge with the sources for the next product release.

If you plan to ship a single international executable, you really cannot postpone fixing bugs related to world-readiness. Developers of localized editions might need a few extra weeks after the domestic edition ships to make changes to resource files, Help files, or sample documents; but once your single executable is done, it should be done for all languages. Of course, this won't always happen, but striving for this ideal will definitely benefit the ship dates of your localized editions.


Summary

Planning and designing a software project that is world-ready will require foresight and an ability to see the big global picture. As you plan and take steps to create a single-binary, world-ready product, educate your team about internationalization issues and hold all team members responsible for the internationalization of the product. Make sure you get international input on the product design early in the process, preferably during the specification stage. Pay attention to legal issues that might affect your ability to sell your product internationally.

Create a versatile user interface and code design that make customization easy, as some features will be necessary for some locales but not for others. As part of the globalization process, design your code to handle international issues pertaining to data input, output, display, and encoding. Be aware of accessibility issues, many of which overlap with world-readiness issues.

In addition, use one set of source-code files to create all language editions of your product. Eliminate from your design any compile dependencies such as #ifdef statements, hard-coded strings or constants, and localizable resources in header files. Never use #ifdef statements to handle cases specific to individual languages. Also, steer clear of coding assumptions that are based on a single language (for example, that words are always separated by spaces, that characters are always Latin, or that the text stream is always left-to-right).

Help translators work more efficiently by:

  • Making your code easier to localize (localizability).
  • Using comments to document strings.
  • Allowing translators to compile and check their work.

Finally, test international functionality early in the development cycle, and do not wait until after the domestic product is completed to fix international-related bugs. By closely following the practices discussed in this chapter, your software project-just like a carefully planned, ideal community-will reap the rewards.


Part II: Globalization

Now that you have seen the importance of creating internationalized products, and learned about the steps inherent to the design process, it is time to examine one of the building blocks of world-ready software—globalization. As mentioned in Chapter 1, "Understanding Internationalization," creating software for the global market consists of both world-readiness and localization, with world-readiness broken down further into globalization and localizability.

While Part I, "Introduction," provided a general conceptual framework with regard to creating and designing world-ready software, including an initial look at globalization and localizability, Part II contains chapters with specific technical details to help you globalize your product. There are eight tasks necessary to achieving this goal. The first six tasks—shown in the following list—are discussed in Part II, while the final two are discussed in Part III,"Localizability."

  1. Write fully implemented Unicode applications. (See Chapter 3, "Unicode." )
  2. Don't make locale-specific assumptions when displaying data. (See Chapter 4, "Locale and Cultural Awareness." )
  3. Handle different languages and methods for entering text. (See Chapter 5, "Text Input, Output, and Display.")
  4. Don't depend on a given font. (See Chapter 5.)
  5. Be aware that you might need to handle complex scripts. (See Chapter 5.)
  6. Allow the user to select the language of the user interface (UI). (See Chapter 6, "Multilingual User Interface [MUI].")
  7. Create an easy-to-localize UI (localizability). (See Part III-Chapter 7, "Software Localizability." )
  8. Make sure your UI is mirroring-aware for bidirectional languages. (See Part III, Chapter 8, "Mirroring." )

Note that the chapters in Part II deal with some rather advanced concepts of coding Microsoft Win32 applications and of creating Web content. Therefore, it is assumed that you are already familiar with Win32 and Web development coding, and that you want to expand your scope of knowledge to encompass world-readiness.


Chapter 3: Unicode
Glossary
  • Character encoding: A method or system of assigning numeric values to characters (for example, ASCII, Unicode, Windows 1252).
  • Unicode: A worldwide character encoding that includes most of the world's scripts; it is developed, maintained and promoted by the Unicode Consortium, a nonprofit computer industry organization. (The official Unicode Consortium Web site is http://www.unicode.org.)
  • The .NET Framework: A platform that enables the creation and use of Extensible Markup Language (XML)-enabled applications, processes, and Web sites as services that share and combine information and functionality with each other by design. The Microsoft .NET Framework is available for Windows platforms or smart devices, providing tailored solutions for organizations and individuals.

Imagine a room filled with people gathered from varying nations, where each individual is speaking a language that is basically unintelligible to the others in the room. In the midst of this confusion, however, each person has been charged with communicating a vital message to the others. In a sense, dealing with varying encoding standards in some ways bears resemblance to this scenario. Most developers of international programs have at some point experienced frustration when trying to work with character encodings. The mishmash of standards makes it hard for users to share data and for programmers to create world-ready software, but Unicode can help combat this problem.

This chapter will begin by helping you understand what Unicode is, its origin, and its purpose as an encoding method. By first looking at traditional encoding methods and some of the problems associated with them, you'll see why Unicode offers a viable and practical solution when you're dealing with multiple scripts and languages, which of course is the case when creating world-ready software. Later subsections will discuss creating Microsoft Win32 Unicode applications, as well as encoding in Web pages, in the .NET Framework, and in console or text-mode programs.

As discussed in Part I, "Introduction," one of the first tasks connected with creating globalized applications is to write Unicode applications. The advantages of using Unicode are many. (See "Unicode's Capabilities" later in this chapter.) Among them, Unicode solves the issue just mentioned of multiple encoding standards. For instance, some standards are 7-bit; others are 8-bit. Single-byte character sets come in several varieties, as do the double-byte standards (which are also called "multibyte" because they are really a mix of single-byte and double-byte character codes). Trying to pass data between different character encodings across networks, platforms, or between operating systems involves a gauntlet of mappings, and headaches.

In the international arena, the ability to share information from a variety of writing systems in a straightforward manner will be increasingly important, especially for applications such as large databases. Take, for instance, a hypothetical European agency based in Belgium that wants to set up a directory to communicate with its French, Greek, Hungarian, and Russian clients. If the company's only computer runs the French edition of Microsoft Windows Millennium Edition (Me), which is based on the Windows 1252 code page for Western European languages, it does not support the Greek and Russian alphabets or certain Hungarian accented characters. Some names will then have to be romanized and others will be spelled with whatever characters are available. In the past this might have been acceptable, but today it cannot be. People want their names spelled correctly, and online transactions require them to be spelled consistently. It's difficult to retrieve archived information using a name that has been transliterated in a dozen different ways. With Unicode, it's easy to share and retrieve information from various scripts and among differing operating systems. This is certainly not the case with traditional character encoding, as the next section illustrates.


Traditional Character Encoding

The mechanism of assigning a code point to a given character-or encoding-has taken various forms before the emergence of Unicode. The following information provides a background on character encoding itself, including single-byte character sets as well as multibyte character sets, both of which preceded Unicode.

Basics of Character Encoding

Glossary
  • ASCII: Acronym for American Standard Code for Information Interchange, a 7-bit encoding. Although primitive, ASCII's set of 128 characters is the one common denominator contained in most of the other standard character sets (and in all Windows and OEM code pages).
  • Code point or code element: (1) The minimum bit combination that can represent a unit of encoded text for processing or exchange. (2) An index into a code page or a Unicode Standard.
  • Character: (1) The smallest components of a writing system or script that have semantic value. A character refers to an abstract idea rather than to a specific glyph or shape that a character might have once rendered or displayed. (2) A code element.

In the past, most computers "spoke" mainly English, with each character represented by a unique numeric value or code point. As shown here, to encode the whole set of characters needed to represent the English language as well as several control characters, a total of 128 distinct code points was needed.

Uppercase (A-Z)                               26
Lowercase (a-z)                               26
Digits (0-9)                                  10
Punctuation marks (. , + { ) %)               32
Space                                          1
Control characters (Tab, CR, LF, and so on)   33

Total                                        128

In the computer world where numeric values are represented by binary data, representing 128 different configurations requires 7 bits to represent each character (27 = 128). The 7-bit encoding system was then created, which was known as "ASCII." (See Figure 3-1.)

Figure 3.1 The ASCII code page.
Figure 3.1 - The ASCII code page.

As computers became more widely used and as additional languages had to be accommodated, support for more comprehensive code pages that included different accented characters-an integral part of many languages-had to be added. As a result, support for new encoding systems such as single-byte character sets (SBCSs) and double-byte character sets (DBCSs) was introduced (each of which will be discussed in the sections that follow).

Single-Byte Character Sets

Glossary
  • Extended characters: (1) Characters above the ASCII range (32 through 127) in single-byte character sets. (2) Accented characters.
  • ANSI: Acronym for the American National Standards Institute. The term "ANSI" as used to signify Windows code pages is a historical reference, but is nowadays a misnomer that continues to persist in the Windows community. The source of this comes from the fact that the Windows code page 1252 was originally based on an ANSI draft-which became International Organization for Standardization (ISO) Standard 8859-1. "ANSI applications" are usually a reference to non-Unicode or code page-based applications.
  • Single-byte character set (SBCS): A character encoding in which each character is represented by 1 byte. Single-byte character sets are mathematically limited to 256 characters.
  • EBCDIC: Extended Binary Coded Decimal Interchange Code. These types of code pages are used on IBM and other manufacturers' mainframes.

With a quick look at Figure 3-1, you can immediately see that the ASCII encoding system's 128-character limitation prevents representation of most languages other than English-including some Latin-based languages that use accented characters, such as French or German. To overcome this limitation, several companies and standards bodies created new code pages using a whole byte (or 8 bits) to represent each character. In each of these new code pages, the set of characters encoded 32 through 127 (hexadecimal 0x20 through 0x7F) was left intact, forming the original 7-bit set called "ASCII". Meanwhile, the characters numbered 128 through 255 (hexadecimal 0x80 through 0xFF) were called "extended characters" and varied from one code page to another. Similarly, this set of extended characters is unique to a particular combination of languages. Therefore, distinct code pages were created to support several Latin-based Western European languages, as well as Greek, Arabic, Turkic, and other additional languages. (See Figure 3-2.)

Figure 3.2 Extended characters and code page.
Figure 3.2 - Extended characters and code page.

As if things were not complicated enough, different implementations of encoding systems were created for the same set of characters! For example, the Latin script for Western European countries has all the following encoding systems:

  • Windows 1252, used primarily in Windows operating systems. (See different Windows encoding tables online at http://www.microsoft.com/globaldev/reference/WinCP.asp.)
  • OEM-850, used in MS-DOS. (See different OEM encoding tables online at http://www.microsoft.com/globaldev/reference/oem.asp.)
  • ISO 8859-1, ISO standard. (See different ISO encoding tables online at http://www.microsoft.com/globaldev/reference/iso.asp.)
  • Several forms of EBCDIC such as EBCDIC-US.
  • MacRoman, used by Apple Computer. (See Macintosh encoding tables online at http://www.microsoft.com/globaldev/dis_v1/html/S2589.asp.)

There are a few points to consider when handling SBCS strings of characters. First, recall that extended characters are different from one code page to another. Therefore your application's behavior is seriously jeopardized if it runs on a system that uses a different code page than what your application is expecting. Code points that you intended for representing one script will be presented and processed as a completely different script, thus displaying gibberish and incorrect results. For example, notice how different the extended characters encoded in the range x80-xFF of the Thai code page 874 (see Figure 3-3) are from the same range of characters encoded in the Windows 1252 code page.

Figure 3.3 Microsoft Windows 874 code page for Thai.
Figure 3.3 - Microsoft Windows 874 code page for Thai.

Second, be aware that certain string manipulation functions may be English-centric. For instance, the string-manipulation code for uppercasing, as shown in the example below, works only for English.

#define ToUpper(ch) ((ch)<='Z' ? (ch) : (ch)+'A' - 'a')

This particular code is English-centric because the assumed relationship between lowercase and uppercase characters is not applicable to accented characters. Also, the meaning of uppercasing is irrelevant for scripts such as Arabic, Hebrew, and East Asian languages because there is no such concept of upper and lowercase in these scripts.

Probably the SBCS's most problematic limitation is its inability to encode East Asian characters and ideographs, since their amount exceeds, by far, the 256 different possible values of an 8-bit encoding. This particular limitation of SBCS encodings has been the prime driving force behind the creation of multibyte character sets.

Multibyte Character Sets

Glossary
  • Multibyte character set (MBCS): A character encoding in which the code points can be either 1, 2, or more bytes.
  • Double-byte character set (DBCS): A character encoding in which the code points can be either 1 or 2 bytes. Used, for example, to encode Chinese, Japanese, and Korean languages.
  • Full-width character: Characters whose glyph image extends across the entire character display cell. In legacy character sets, full-width characters are normally encoded in 2 or more bytes. The Japanese term for full-width characters is "zenkaku."
  • Half-width character : Characters whose glyph image occupies half of the character display cell. In legacy character sets, half-width characters are normally encoded in a single byte. The Japanese term for half-width characters is "hankaku."
  • Ideographic character: A character of Chinese origin representing a word or a syllable that is generally used in more than one Asian language. Sometimes referred to as a "Chinese character."
  • Kana: The set of Japanese hiragana and katakana characters.
  • Shift-JIS: The multibyte encoding developed by Microsoft for Japanese that is based on Japan Industry Standard (JIS) standard X 0208. The name comes from the way the lead bytes in Shift-JIS shift around the encoding range of half-width katakana in JIS X 0208.
  • Lead byte: The first byte of a 2-byte code point in a DBCS code page.
  • Trail byte: The second byte of a 2-byte code point in a DBCS code page.
  • Big-5: The multibyte encoding for Traditional Chinese characters standardized by Taiwan.

A multibyte character set expands the range of encoding possibilities for East Asian languages. For instance, the Chinese script is made up of more than 10,000 basic ideographic characters-called "hantsu" or "hanzi" in Chinese. Corresponding ideographic characters are called "kanji" in Japanese and "hanja" in Korean. As you've seen, single-byte encodings are too limited to accommodate these languages. Once you have assigned 256 code points (that is, indexes to the code page), what do you do with the rest of the several thousand characters that Chinese or Japanese readers and writers typically use every day?

A solution commonly used on computers is to encode most characters (primarily ideographs) with 2-byte values, thus making room for far more than 256 characters. The key element in the previous sentence is "most characters"-characters such as those in the ASCII set and the Japanese phonetic syllabary known as "katakana" still have single-byte representations. The result is a code page that mixes single-byte and double-byte characters.

On other operating systems, such as UNIX, characters can be represented by as many as 4 bytes. (See Figure 3-4.) Character sets that mix character codes of 1, 2, and 3 bytes are generally called "multibyte character sets."

Figure 3.4 Multibyte character sets.
Figure 3.4 - Multibyte character sets.

The mixing of byte lengths in multibyte character sets used in East Asian scripts leads to more complex string-parsing code, where each byte does not necessarily represent a single character. For this reason, you must exercise caution when coding string-parsing routines. Each 2-byte character is composed of a lead byte and a trail byte that must be stored together and manipulated as a unit. A lead-byte value always falls into one or more ranges above 127; no 7-bit ASCII character can be a lead byte. NULL can never be a trail byte but the range of possible trail bytes can overlap to some degree with ASCII. Trail-byte values are frequently indistinguishable from lead-byte values; the only way to tell the difference is from the context of the surrounding characters. Furthermore, most trail bytes taken without their lead byte can be mistaken for a single-byte character. In the previous figure, the kanji character [image] is represented by the byte pair 0x89-0x40, where 0x89 is the lead byte and 0x40-which is the @ sign by itself-is the trail byte. Figure 3-5, meanwhile, shows how lead bytes from the main page map to an extended table of trail bytes that have the actual encoding for kanji characters.

Figure 3.5 Lead-byte and trail-byte mapping in Windows code page 932.
Figure 3.5 - Lead-byte and trail-byte mapping in Windows code page 932.

Special coding techniques specific to DBCSs or MBCSs will not be covered in detail in this book, since the idea of a fully globalized application is to have a Unicode encoding system. However, you'll need to consider the following when dealing with MBCS:

  • Always handle one lead byte and its trail byte as a unit. A lead byte or a trail byte alone does not contain enough information to identify a character.
  • Lead-byte and trail-byte ranges differ from one DBCS code page to another. (See Table 3-1 for an example involving various East Asian languages.) Instead of hard-coding these ranges, you can use Win32 application program ming interfaces (APIs) such as IsDBCSLeadByteEx or GetCPInfoEx to find out if a given byte can be a lead byte in the target code page.
  • Make sure that your string parsing takes into consideration a mixture of SBCS and DBCS. Instead of using pointer incrementals or decrementals, it's suggested you use predefined APIs such as CharPrevExA and CharNextExA to go to the previous or next character.
  • Caret positioning and cursor movements are common issues when dealing with MBCS. Not only do coding practices need to be adjusted to avoid splitting double-byte characters in two, but so do a program's display operations. Rules of selection, cursor placement, and cursor movement are the same as you would expect when dealing with alphabetic characters-the cursor should always end up between characters and never in the middle of one. The difference with double-byte characters is that they are a combination of two encoding units.

  • (For more information on MBCS encoding, see the first edition of this book. Or go to http://www.microsoft.com/globaldev/dis_v1/html/S24B2_b1.asp.)

Table 3-1 includes the lead-byte and trail-byte ranges for the code pages corresponding to various East Asian languages.

Table 3-1 Lead-byte and trail-byte ranges for some code pages used with East Asian languages.
Language | Character Set Name | Code Page | Lead-Byte Ranges | Trail-Byte Ranges
Chinese (Simplified) | GB 2312-80 | CP 936 | 0xA1-0xFE | 0xA1-0xFE
Chinese (Traditional) | Big-5 | CP 950 | 0x81-0xFE | 0x40-0x7E 0xA1-0xFE
Japanese | Shift-JIS (Japan Industry Standard) | CP 932 | 0x81-0x9F 0xE0-0xFC | 0x40-0xFC (except 0x7F)
Korean (Wansung) | KS C-5601-1987 | CP 949 | 0x81-0xFE | 0x41-0x5A 0x61-0x7A 0x81-0xFE

Once again, though the current shift is toward utilizing the Unicode encoding system, there is one final consideration should you need to work with code pages. Before developers started using Unicode extensively, Microsoft had a full array of Win32 APIs to make working with code pages easier. (You can find more information about these APIs at http://msdn.microsoft.com.) In fact, in order to deal with non-Unicode-based text strings, Microsoft still supports these APIs, which include the following. (See Table 3-2.)

Table 3-2 Win32 APIs used for working with code pages, before the extensive use of Unicode.
API | Function
AreFileApisANSI | Determines whether the file I/O functions are using the Windows or OEM character-set code page
EnumSystemCodePages | Enumerates the system code pages (including OEM, Windows, and EBCDIC) that are either installed or supported, depending on the specified input arguments
GetACP | Returns currently selected Windows code page
GetConsoleCP | Returns the code-page ID of the current console code page
GetConsoleOutputCP | Returns the code-page ID of the current console output code page
GetCPInfoEx | Returns a structure containing the maximum width of a character in the specified code page; the lead-byte ranges, if any; and the code page's default character, used in conjunction with WideCharToMultiByte
GetOEMCP | Returns currently selected OEM code page
GetSystemDefaultLangID | Retrieves the language identifier of the system locale
GetSystemDefaultLCID | Retrieves the system default locale identifier
IsDBCSLeadByteEx | Determines whether a character is a lead byte in the specified code page
IsValidCodePage | Determines whether a code page is installed in the system
MultiByteToWideChar | Maps a character string to a wide-character (UTF-16) string; the character string mapped by this function is not necessarily from a multibyte character set
SetConsoleCP | Sets the code-page ID of the current console code page
SetConsoleOutputCP | Sets the code-page ID of the current console output code page
SetFileApisToANSI | Causes the file I/O functions to use the Windows character-set code page
SetFileApisToOEM | Causes the file I/O functions to use the OEM character- set code page
WideCharToMultiByte | Maps a wide-character (UTF-16) string to a new character string; the new character string is not necessarily from a multibyte character set

Because of the limitations of SBCS and MBCS, however, and because of the complexity of properly handling MBCS character strings, several leading Information Technology (IT) companies jointly created the Unicode Consortium. (See "Unicode's Capabilities" later in this chapter.) The resulting universal system known as "Unicode" is helping to standardize character encoding and to make multilingual computing a reality. In the sections that follow, you'll see the possibilities that Unicode offers and gain a better understanding of how it works.


A Closer Look at Unicode

As you've seen by now, with the emergence of the Unicode Standard you can consistently handle English text as well as East Asian and other major scripts of the world. This ability is essential when dealing with an increasingly global market. The following sections will first give you a very brief look at Unicode's origin, and will then show you Unicode's capabilities. You'll also gain a deeper understanding of Unicode-aware functions. The discussion will then switch to a more technical nature, with specific details on how to create Win32 Unicode applications, as well as how to use encodings in Web pages, in the .NET Framework, and finally, in console or text-mode programs.

Unicode's Capabilities

Glossary
  • Base character: An encoding code point that does not graphically combine with preceding characters, and that is neither a control nor a format character. The Latin "a" is an example of a base character.
  • Combining character: A character that graphically combines with a preceding base character. The combining acute accent mark ( [image])-U+0301-is an example of a combining character.
  • Precomposed character: A character that is equivalent to a sequence of one or more characters. It is also known as a "composed character" or a "composite character." Thus the combining character sequence "a + ' " forms the precomposed, composed, or composite character "à."
  • ISO 10646: The International Organization for Standardization's encoding that is code-for-code equivalent to Unicode.
  • Private Use Area: The area in Unicode repertoire from U+E000 through U+F8FF, U+F0000 through U+FFFFD, and U+100000 through U+10FFFD that is set aside for vendor-specific or user-designed characters.
  • Contextual analysis: A process for determining how to handle text based on surrounding characters, as in Arabic, in which a glyph changes shape depending on its position in a word.
  • Glyph: The actual shape (bit pattern, outline, and so forth) of a character image. For example, an italic "a " and a roman "a" are two different glyphs representing the same underlying character.
  • Logical order: In the same order in which it is typed. Generally refers to text that might be displayed in a different order, such as Arabic, Hebrew, or bidirectional text.
  • Neutral character: A character whose directionality (right-to-left or left-to right) is dependent on the directionality of the characters that surround it. See contextual analysis.
  • Request For Comments (RFC) documents: The written definitions of Internet protocols and policies.
  • Little-endian: A computer architecture that stores multibyte numerical values with the least significant byte values first. On systems using little-endian architecture, the letter "A" (U+0041) is stored as 0x41 0x00.
  • Big-endian: A computer architecture that stores multibyte numerical values with the most significant byte values first. On systems using big-endian architecture, the letter "A" (U+0041) is stored as 0x00 0x41.

The complex programming methods required for working with mixed-byte encodings, the involved process of creating new code pages every time another language requires computer support, and the importance of mixing and sharing information in a variety of languages across different systems were some of the factors motivating the creators of the Unicode encoding standard.

Unicode originated through collaboration between Xerox and Apple. An ad hoc committee of several companies then formed, and others, including IBM and Microsoft, rapidly joined. In 1991, this group founded the Unicode Consortium whose membership now includes several leading Information Technology (IT) companies. (For more information on Unicode, visit the Unicode Consortium's site at http://www.Unicode.org.)

Unicode is an especially good fit for the age of the Internet, since the worldwide nature of the Internet demands solutions that work in any language. The World Wide Web Consortium (W3C) has recognized this fact and now expects all new RFCs to use Unicode for text. Many other products and standards now require or allow use of Unicode; for example, XML, HTML, Microsoft JScript, Java, Perl, Microsoft C#, and Microsoft Visual Basic 7 (VB.NET). Today, Unicode is the de facto character encoding standard accepted by all major computer companies, while ISO 10646 is the corresponding worldwide de jure standard approved by all ISO member countries. The two standards include identical character repertoires and binary representations.

Unicode encompasses virtually all characters used widely in computers today. It is capable of addressing more than 1.1 million code points. The standard has provisions for 8-bit, 16-bit and 32-bit encoding forms. The 16-bit encoding is used as its default encoding and allows for its million plus code points to be distributed across 17 "planes" with each plane addressing over 65,000 characters each. The characters in Plane 0-or as it is commonly called the "Basic Multilingual Plane" (BMP)-are used to represent most of the world's written scripts, characters used in publishing, mathematical and technical symbols, geometric shapes, basic dingbats (including all level-100 Zapf Dingbats), and punctuation marks. But in addition to the support for characters in modern languages and for the symbols and shapes just mentioned, Unicode also offers coverage for other characters, such as less commonly used Chinese, Japanese, and Korean (CJK) ideographs, Arabic presentation forms, and musical symbols. Many of these additional characters are mapped beyond the original plane using an extension mechanism called "surrogate pairs." With Unicode 3.2, over 95,000 code points have already been assigned characters; the rest have been set aside for future use. Unicode also provides Private Use Areas of over 131,000 locations available to applications for user-defined characters, which typically are rare ideographs representing names of people or places.

Figure 3-6 shows the Unicode encoding layout for the BMP (Plane 0) in abstract form. (To see the entire Unicode code-point allocation, see Appendix G, "Unicode Code-Point Allocation.")

Figure 3.6 Unicode encoding layout for the BMP (Plane 0).
Figure 3.6 - Unicode encoding layout for the BMP (Plane 0).

Unicode rules, however, are strict about code-point assignment-each code point has a distinct representation. There are also many cases in which Unicode deliberately does not provide code points. Variants of existing characters are not given separate code points, because to do so would represent duplicate encoding of what is underlying the same character. Examples are font variants (such as bold and italic) and glyph variants, which basically are different ways of representing the same characters.

For the most part, Unicode defines characters uniquely, but some characters can be combined to form others, such as accented characters. The most common accented characters, which are used in French, German, and many other European languages, exist in their precomposed forms and are assigned code points. These same characters can be expressed by combining a base character with one or more nonspacing diacritic marks. For example, "a" followed by a nonspacing accent mark is displayed as "à." Nonspacing accent marks make it possible to have a large set of accented characters without assigning them all distinct code points. This is useful for representing accented characters in written languages that are less widely used, such as some African languages. It's also useful for creating a variety of mathematical symbols. The precomposed characters are encoded in the Unicode Standard primarily for compatibility with other encodings. The Unicode Standard contains strict rules for determining the equivalence of precomposed characters to combining character sequences. (See Figure 3-7.) The Win32 API function FoldStringW maps multiple combining characters into precomposed forms. Also, MultiByteToWideChar can be used with either the MB_PRECOMPOSED or the MB_COMPOSITE flags for mapping characters to their precomposed or composite forms.

For all their advantages, Unicode Standards are far from a panacea for internationalization. The code-point positions of Unicode elements do not imply a sort order, and Unicode does not encode font information. It is the operating system that defines these rules, as in the case of Win32-based applications, which need to obtain sorting and font information from the operating system.

In addition, basing your software on the Unicode Standard is only one step in the internationalization process. You still need to write code that adapts to cultural preferences or language rules. (For more information on other globalization considerations, see Chapter 4, "Locale and Cultural Awareness," Chapter 5, "Text Input, Output, and Display," and Chapter 6, "Multilingual User Interface [MUI].")

As a further caveat, not all Unicode-based text processing is a matter of simple character-by-character parsing. Complex text-based operations such as hyphenation, line breaking, and glyph formation need to take into account the context in which they are being used (the relation to surrounding characters, for instance). The complexity of these operations hinges on language rules and has nothing to do with Unicode as an encoding standard. Instead, the software implementation should define a higher-level protocol for handling these operations.

In contrast, there are unusual characters that have very specific semantic rules attached to them; these characters are detailed in the The Unicode Standard.1 Some characters always allow a line break (for example, most spaces), whereas others never do (for instance, nonspacing or nonbreaking characters). Still other characters, including many used in Arabic and Hebrew, are defined as having strong or weak text directionality. The Unicode Standard defines an algorithm for determining the display order of bidirectional text, and it also defines sev eral "directional formatting codes" as overrides for cases not handled by the implicit bidirectional ordering rules to help create comprehensible bidirectional text. These formatting codes allow characters to be stored in logical order but displayed appropriately depending on their directionality. Neutral characters, such as punctuation marks, assume the directionality of the strong or weak characters nearby. Formatting codes can be used to delineate embedded text or to specify the directionality of characters. (For more information on displaying bidirectional Unicode-based text, see The Unicode Standard.)

Figure 3.7 Precomposed and composite characters.
Figure 3.7 - Precomposed and composite characters.

Unicode's Functionality

Glossary
  • Plaintext: Computer-encoded text that contains only code elements and no other formatting or structural information (for example, font size, font type, or other layout information). Plaintext exchange is commonly used between computer systems that might have no other way to exchange information.
  • Wide character: A character encoded by a wchar_t or a 16-bit (WORD) data type. Often used to refer to UTF-16-encoded characters.

You've now seen some of the capabilities that Unicode offers. The sections that follow delve deeper into Unicode's functions to provide helpful information as you work with Unicode Standards and encodings. For instance, what is the function of byte-order marks (BOMs)? What are surrogate pairs, and how do they enable you to go from encoding 65,000 characters to over 1 million additional characters? These and other questions will be explored in the following sections.

Transformations of Unicode Code Points

There are different techniques to represent each one of the Unicode code points in binary format. Each of the following techniques uses a different mapping to represent unique Unicode characters. The Unicode encodings are:

  • UTF-8: To meet the requirements of byte-oriented and ASCII-based systems, UTF-8 has been defined by the Unicode Standard. Each character is represented in UTF-8 as a sequence of up to 4 bytes, where the first byte indicates the number of bytes to follow in a multibyte sequence, allowing for efficient string parsing. UTF-8 is commonly used in transmission via Internet protocols and in Web content.
  • UTF-16: This is the 16-bit encoding form of the Unicode Standard where characters are assigned a unique 16-bit value, with the exception of characters encoded by surrogate pairs, which consist of a pair of 16-bit values. The Unicode 16-bit encoding form is identical to the International Organization for Standardization/International Electrotechnical Commision (ISO/IEC) transformation format UTF-16. In UTF-16, any characters that are mapped up to the number 65,535 are encoded as a single 16-bit value; characters mapped above the number 65,535 are encoded as pairs of 16-bit values. (For more information on surrogate pairs, see "Surrogate Pairs" later in this chapter.) UTF-16 little-endian is the encoding standard at Microsoft (and in the Windows operating system).
  • UTF-32: Each character is represented as a single 32-bit integer.

Figure 3-8 shows two characters encoded in both code pages and Unicode, using UTF-16 and UTF-8.

Figure 3.8 The character A and a kanji character encoded in code pages and in Unicode with both UTF-16 and UTF-8.
Figure 3.8 - The character "A" and a kanji character encoded in code pages and in Unicode with both UTF-16 and UTF-8.

Since UTF-8 is so commonly used in Web content, it's helpful to know how Unicode code points get mapped into this encoding without introducing the hassle of MBCS characters. Table 3-3 shows the relationship between Unicode code points and a UTF-8-encoded character. The starting byte of a chain of bytes in a UTF-8 encoded character tells how many bytes are used to encode that character. All the following bytes start with the mark "10" and the xxx's denote the binary representation of the encoding within the given range.

Table 3-3 Relationship between Unicode code points and a UTF-8-encoded character. In UTF-8, the first byte indicates the number of bytes to follow in a multibyte-encoded sequence.

Unicode Range | UTF-8-Encoded Bytes
0x00000000-0x0000007F | 0 xxxxxxx
0x00000080-0x000007FF | 110 xxxxx 10 xxxxxx
0x00000800-0x0000FFFF | 1110 xxxx 10 xxxxxx 10 xxxxxx
0x00010000-0x001FFFFF | 11110 xxx 10 xxxxxx 10 xxxxxx 10 xxxxxx

Using this approach, once encoded in UTF-8, [image] would become 0xE9 0x99 0xA2, which in binary is 0x11101001 0x10011001 0x10100010.

Byte-Order Marks

Another concept to be familiar with as you work with Unicode is that of byte- order marks. A BOM is used to indicate how a processor places serialized text into a sequence of bytes. If the least significant byte is placed in the initial position, this is referred to as "little-endian," whereas if the most significant byte is placed in the initial position, the method is known as "big-endian." A BOM can also be used as a reference to identify the encoding of the text file. Notepad, for example, adds the BOM to the beginning of each file, depending on the encoding used in saving the file. This signature will allow Notepad to reopen the file later. Table 3-4 shows byte-order marks for various encodings. The UTF-8 BOM identifies the encoding format rather than the BOM of the document-since each character is represented by a sequence of bytes.

Table 3-4 Binary representation of the byte-order mark (U+FEFF) for specific encodings.

Encoding | Encoded BOM
UTF-16 big-endian | FE FF
UTF-16 little-endian | FF FE
UTF-8 | EF BB BF

Surrogate Pairs

With the Unicode 16-bit encoding system, over 65,000 characters can be encoded (216 = 65536). However, the total number of characters that needs to be encoded has actually exceeded that limit (mainly to accommodate the CJK extension of characters). To find additional place for new characters, developers of the Unicode Standard decided to introduce the notion of surrogate pairs. With surrogate pairs, a Unicode code point from range U+D800 to U+DBFF (called "high surrogate") gets combined with another Unicode code point from range U+DC00 to U+DFFF (called "low surrogate") to generate a whole new character, allowing the encoding of over 1 million additional characters. Unlike MBCS characters, high and low surrogates cannot be interpreted when they do not appear as part of a surrogate pair (one of the major challenges with lead-byte and trail-byte processing of MBCS text).

For the first time, in Unicode 3.01 characters are encoded beyond the original 16-bit code space or the BMP (Plane 0). These new characters, encoded at code positions of U+10000 or higher, are synchronized with the international standard ISO/IEC 10646-2. In addition to two Private Use Areas-plane 15 (U+F0000 - U+FFFFD) and plane 16 (U+100000 - U+10FFFD)-Unicode 3.1 and 10646-2 define three new supplementary planes:

  • Supplementary Multilingual Plane (SMP)-with code positions from U+10000 through U+1FFFF
  • Supplementary Ideographic Plane (SIP)-with code positions from U+20000 through U+2FFFF
  • Supplementary Special-purpose Plane (SSP)-with code positions from (SSP) U+E0000 through U+EFFFF

The SMP, or Plane 1, contains several historic scripts and several sets of symbols: Old Italic, Gothic, Deseret, Byzantine Musical Symbols, (Western) Musical Symbols, and Mathematical Alphanumeric Symbols. Together these comprise 1,594 newly encoded characters. The SIP, or Plane 2, contains a very large collection of additional unified Han ideographs known as "CJK Extension B," comprising 42,711 characters, as well as 542 additional CJK Compatibility ideographs. The SSP, or Plane 14, contains a set of 97 tag characters.

Windows Support for Unicode

You've now learned more about the benefits and capabilities that Unicode offers, in addition to looking more closely at its functionality. You might also be wondering about the extent to which Windows supports Unicode's features. Microsoft Windows NT 3.1 was the first major operating system to support Unicode, and since then Microsoft Windows NT 4, Microsoft Windows 2000, and Microsoft Windows XP have extended this support, with Unicode being their native encoding. In fact, when you run a non-Unicode application on them, the operating system converts the application's text internally to Unicode before any processing is done. The operating system then converts the text back to the expected code-page encoding before passing the information back to the application.

In addition, Windows XP supports a majority of the Unicode code points with fonts, keyboard drivers, and other system files necessary to the input and display of content in all supported languages. Once again, the fundamental representation of text in Windows NT-based operating systems is UTF-16, and the WCHAR data type is a UTF-16 code unit. Windows does provide interfaces for other encodings in order to be backward-compatible, but converts such text to UTF-16 internally. The system also provides interfaces to convert between UTF-16 and UTF-8 and to inquire about the basic properties of a UTF-16 code point (for example, whether it is a letter, a digit, or a punctuation mark). Since Microsoft Windows 95, Microsoft Windows 98, and Windows Me are not Unicode-based, they provide only a small subset of the Unicode support available in the Windows NT-based versions of Windows. Thus by working with Unicode and Windows NT-based operating systems, you are yet one step closer toward the goal of creating world-ready applications. The remaining sections will show you practical techniques and examples for creating Win32 Unicode applications, as well as tips for using encoding for Web pages, in the .NET Framework, and in console or text-mode programming.


Creating Win32 Unicode Applications

Glossary
  • System locale: The system locale determines the Windows code page used by the ANSI (non-Unicode) version of Win32 APIs. String and character parameters passed to a Win32 ANSI are converted from this Windows code page to Unicode.
  • Win32 API: The set of 32-bit functions supported by Windows.

You've decided to create a Win32 Unicode application. Following the tips and examples included here will help you to get underway and to perform your tasks efficiently-whether you're converting between code-page encodings and Unicode, compiling your Unicode application in Microsoft Visual C++, migrating your code page-based application to Unicode, or writing Unicode code. Although adopting the Unicode encoding system for your application is only a first step toward fully globalized software, it certainly is the most important factor in multilingual computing. In fact, any language version of a Unicode Windows application can run on any language version of Windows 2000 and Windows XP, regardless of the user's settings or the language version of the operating system.

As mentioned earlier, Windows 2000 and Windows XP use Unicode as their default encoding system. Whenever dealing with data based on code pages, the data is internally converted to UTF-16 and processed in Unicode. This means that a legacy or non-Unicode application's behavior depends on the system's conversion from code-page encoding to Unicode. The system performs this conversion based on the value of the default code page of what is called the system locale. Windows XP refers to this variable as "Language for non-Unicode programs" (found when clicking the Advanced tab within the Regional And Language Options property sheet). (See Figure 3-9.) For each system, this value can only be changed or set by a system administrator and requires the computer to be restarted. The primary effect of the system-locale value is to set the internal system's conversion from the code page to Unicode and vice versa.

Figure 3.9 Windows XP Regional And Language Options property sheet, Advanced tab.
Figure 3.9 - Windows XP Regional And Language Options property sheet, Advanced tab.

In Figure 3-9, the system locale is set to Russian (Windows code page 1251). This means that if you are not dealing with a Unicode application, the default text encoding in the application has to be either 1251 or ASCII. All applications built on other encodings will fail, since the system will be using the Russian code page to convert between the code page and Unicode, unless they implement their own code-page conversion mechanism. Figure 3-10 shows how a message box from a non-Unicode Arabic application running with an English system locale would appear. The conversion between code page and Unicode fails to map to appropriate Arabic characters, and instead randomly maps to extended Windows 1252 characters.

Figure 3.10 Non-Unicode application running on mismatched system locales.
Figure 3.10 - Non-Unicode application running on mismatched system locales.

Porting existing code page-based applications to Unicode is easier than you might think. In fact, Unicode was implemented in such a way as to make writing Unicode applications almost transparent to developers. Unicode also needed to be implemented in such a way as to ensure that non-Unicode applications remain functional whenever running in a pure Unicode platform. To accommodate these needs, the implementation of Unicode required changes in two major areas:

  • Creation of a data-type variable (WCHAR) to handle 16-bit characters
  • Creation of a set of APIs that accept string parameters with 16-bit character encoding

WCHAR, a 16-Bit Data Type

Most string operations for Unicode can be coded with the same logic used for handling the Windows character set. The difference is that the basic unit of operation is a 16-bit quantity instead of an 8-bit one. The header files provide a number of type definitions that make it easy to create sources that can be compiled for Unicode or the Windows character set.

For 8-bit (ANSI) and double-byte characters:
    typedef char CHAR;            // 8-bit character
    typedef char *LPSTR;         // pointer to 8-bit string
 For Unicode (wide) characters:
    typedef unsigned short WCHAR; // 16-bit character
    typedef WCHAR *LPWSTR; // pointer to 16-bit string

Figure 3-11 shows the method by which the Win32 header files define three sets of types:

  • One set of generic type definitions (TCHAR, LPTSTR), which depend on the state of the _UNICODE manifest constant.
  • Two sets of explicit type definitions (one set for those that are based on code pages or ANSI and one set for Unicode).

With generic declarations, it is possible to maintain a single set of source files and compile them for either Unicode or ANSI support.

Figure 3.11 WCHAR, a new data type.
Figure 3.11 - WCHAR, a new data type.

W Function Prototypes for Win32 APIs

All Win32 APIs that take a text argument either as an input or output variable have been provided with a generic function prototype and two definitions: a version that is based on code pages or ANSI (called "A") to handle code page-based text argument and a wide version (called "W ") to handle Unicode. The generic function prototype consists of the standard API function name implemented as a macro. The generic prototype gets resolved into one of the explicit function prototypes ("A " or "W "), depending on whether the compile-time manifest constant UNICODE is defined in a #define statement. The letter "W" or "A" is added at the end of the API function name in each explicit function prototype.

// windows.h
#ifdef UNICODE
#define SetWindowText SetWindowTextW
#else
#define SetWindowText SetWindowTextA
#endif // UNICODE

With this mechanism, an application can use the generic API function to work transparently with Unicode, depending on the #define UNICODE manifest constant. It can also make mixed calls by using the explicit function name with "W" or "A."

One function of particular importance in this dual compile design is RegisterClass (and RegisterClassEx). A window class is implemented by a window procedure. A window procedure can be registered with either the RegisterClassA or RegisterClassW function. By using the function's "A" version, the program tells the system that the window procedure of the created class expects messages with text or parameters that are based on code pages; other objects associated with the window are created using a Windows code page as the encoding of the text. By registering the window class with a call to the wide-character version of the function, the program can request that the system pass text parameters of messages as Unicode. The IsWindowUnicode function allows programs to query the nature of each window.

On Windows NT 4, Windows 2000, and Windows XP, "A" routines are wrappers that convert text that is based on code pages or ANSI to Unicode-using the system-locale code page-and that then call the corresponding "W" routine. On Windows 95, Windows 98, and Windows Me, "A" routines are native, and most "W" routines are not implemented. If a "W" routine is called and yet not implemented, the ERROR_CALL_NOT_ IMPLEMENTED error message is returned. (For more information on how to write Unicode-based applications for non-Unicode platforms, see Chapter 18, "Microsoft Layer for Unicode [MSLU].")

Unicode Text Macro

Visual C++ lets you prefix a literal with an "L" to indicate it is a Unicode string, as shown here:

 LPWSTR str = L"This is a Unicode string";

In the source file, the string is expressed in the code page that the editor or compiler understands. When compiled, the characters are converted to Unicode. The Win32 SDK resource compiler also supports the "L" prefix notation, even though it can interpret Unicode source files directly. WINDOWS.H defines a macro called TEXT() that will mark string literals as Unicode, depending on whether the UNICODE compile flag is set.

#ifdef UNICODE
#define TEXT(string) L#string
#else
#define TEXT(string) string
#endif // UNICODE

So, the generic version of a string of characters should become:

 LPTSTR str = TEXT("This is a generic string");

C Run-Time Extensions

The Unicode data type is compatible with the wide-character data type wchar_t in ANSI C, thus allowing access to the wide-character string functions. Most of the C run-time (CRT) libraries contain wide-character versions of the strxxx string functions. The wide-character versions of the functions all start with wcs. (See Table 3-5.)

Table 3-5 Examples of C run-time library routines used for string manipulation.

Generic CRT | 8-bit Character Sets | Unicode
_tcscpy | strcpy | wcscpy
_tcscmp | strcmp | wcscmp

The C run-time library also provides such functions as mbtowc and wctomb, which can translate the C character set to and from Unicode. The more general set of functions of the Win32 API can perform the same functions as the C run-time libraries including conversions between Unicode, Windows character sets, and MS-DOS code pages. In Windows programming, it is highly recommended that you use the Win32 APIs instead of the CRT libraries in order to take advantage of locale-aware functionality provided by the system, as described in Chapter 4. (See Table 3-6.)

Table 3-6 Equivalent Win32 API functions for the C run-time library routines found in Table 3-5.
Generic Win32 | 8-bit Character Sets | Unicode
lstrcpy | lstrcpyA | lstrcpyW
lstrcmp | lstrcmpA | lstrcmpW

Conversion Functions Between Code Page and Unicode

Since a large number of applications are still code page-based, and since you might want to support Unicode internally, there are a lot of occasions where a conversion between code-page encodings and Unicode is necessary. The pair of Win32 APIs,MultiByteToWideChar and WideCharToMultiByte, allow you to convert code-page encoding to Unicode and Unicode data to code-page encoding, respectively. Each of these APIs takes as an argument the value of the code page to be used for that conversion. You can, therefore, either specify the value of a given code page (example: 1256 for Arabic) or use predefined flags such as:

  • CP_ACP: for the currently selected system Windows code page.
  • CP_OEMCP: for the currently selected system OEM code page.
  • CP_UTF8: for conversions between UTF-16 and UTF-8.

(For more information, see the Microsoft Developer Network [MSDN] documentation at http://msdn.microsoft.com.)

By using MultiByteToWideChar and WideCharToMultiByte consecutively, using the same code-page information, you do what is called a "round trip." If the code-page number that is used in this encoding conversion is the same as the code-page number that was used in encoding the original string, the round trip should allow you to retrieve the initial character string.

Compiling Unicode Applications in Visual C++

By using the generic data types and function prototypes, you have the liberty of creating a non-Unicode application or compiling your software as Unicode. To compile an application as Unicode in Visual C/C++, go to Project/Settings/C/C++ /General, and include UNICODE and _UNICODE in Preprocessor Definitions. The UNICODE flag is the preprocessor definition for all Win32 APIs and data types, and _UNICODE is the preprocessor definition for C run-time functions.

Migration to Unicode

Glossary
  • Compatibility zone: The area in Unicode repertoire from U+F900 through U+FFEF that is assigned to characters from other standards. These characters are variants of other Unicode characters.

Creating a new program based on Unicode is fairly easy. Unicode has a few features that require special handling, but you can isolate these in your code. Converting an existing program that uses code-page encoding to one that uses Unicode or generic declarations is also straightforward. Here are the steps to follow:

  1. Modify your code to use generic data types. Determine which variables declared as char or char* are text, and not pointers to buffers or binary byte arrays. Change these types to TCHAR and TCHAR*, as defined in the Win32 file WINDOWS.H, or to _TCHAR as defined in the Visual C++ file TCHAR.H. Replace instances of LPSTR and LPCH with LPTSTR and LPTCH. Make sure to check all local variables and return types. Using generic data types is a good transition strategy because you can compile both ANSI and Unicode versions of your program without sacrificing the readability of the code. Don't use generic data types, however, for data that will always be Unicode or always stays in a given code page. For example, one of the string parameters to MultiByteToWideChar and WideCharToMultiByte should always be a code page-based data type, and the other should always be a Unicode data type.
  2. Modify your code to use generic function prototypes. For example, use the C run-time call _tcslen instead of strlen, and use the Win32 API SetWindowText instead of SetWindowTextA. This rule applies to all APIs and C functions that handle text arguments.
  3. Surround any character or string literal with the TEXT macro. The TEXT macro conditionally places an "L" in front of a character literal or a string literal definition. Be careful with escape sequences. For example, the Win32 resource compiler interprets L/" as an escape sequence specifying a 16-bit Unicode double-quote character, not as the beginning of a Unicode string.
  4. Create generic versions of your data structures. Type definitions for string or character fields in structures should resolve correctly based on the UNICODE compile-time flag. If you write your own string-handling and character-handling functions, or functions that take strings as parameters, create Unicode versions of them and define generic prototypes for them.
  5. Change your build process. When you want to build a Unicode version of your application, both the Win32 compile-time flag -DUNICODE and the C run-time compile-time flag -D_UNICODE must be defined.
  6. Adjust pointer arithmetic. Subtracting char* values yields an answer in terms of bytes; subtracting wchar_t* values yields an answer in terms of 16-bit chunks. When determining the number of bytes (for example, when allocating memory for a string), multiply the length of the string in symbols by sizeof(TCHAR). When determining the number of characters from the number of bytes, divide by sizeof(TCHAR). You can also create macros for these two operations, if you prefer. C makes sure that the ++ and -- operators increment and decrement by the size of the data type. Or even better, use Win32 APIs CharNext and CharPrev.
  7. Check for any code that assumes a character is always 1 byte long. Code that assumes a character's value is always less than 256 (for example, code that uses a character value as an index into a table of size 256) must be changed. Make sure your definition of NULL is 16 bits long.
  8. Add code to support special Unicode characters. These include Unicode characters in the compatibility zone, characters in the Private Use Area, combining characters, and characters with directionality. Other special characters include the Private Use Area noncharacter U+FFFF, which can be used as a placeholder, and the byte-order marks U+FEFF and U+FFFE, which can serve as flags that indicate a file is stored in Unicode. The byte-order marks are used to indicate whether a text stream is little-endian or big-endian. In plaintext, the line separator U+2028 marks an unconditional end of line. Inserting a paragraph separator, U+2029, between paragraphs makes it easier to lay out text at different line widths.
  9. Debug your port by enabling your compiler's type-checking. Do this with and without the UNICODE flag defined. Some warnings that you might be able to ignore in the code page-based world will cause problems with Unicode. If your original code compiles cleanly with type-checking turned on, it will be easier to port. The warnings will help you make sure that you are not passing the wrong data type to code that expects wide-character data types. Use the Win32 National Language Support API (NLS API) or equivalent C run-time calls to get character typing and sorting information. Don't try to write your own logic for handling locale-specific type checking-your application will end up carrying very large tables!

In the following example, a string is loaded from the resources and is used in two scenarios:

  • As a body to a message box
  • To be drawn at run time in a given window

(For more information on resources, see Chapter 7, "Software Localizability Guidelines.")

For the purpose of simplification, this example will ignore where and how irrelevant variables have been defined. Suppose you want to migrate the following code page-based code to Unicode:

char g_szTemp[MAX_STR];         // Definition of a char data type
// Loading IDS_SAMPLE from the resources in our char variable
LoadString(g_hInst, IDS_SAMPLE, g_szTemp, MAX_STR);
// Using the loaded string as the body of the message box
MessageBox(NULL, g_szTemp, "This is an ANSI message box!", MB_OK);
// Using the loaded string in a call to TextOut for drawing at
// run time
ExtTextOut(hDC, 10, 10, ETO_CLIPPED , NULL, g_szTemp,
strlen(g_szTemp), NULL);

Migrating this code to Unicode is as easy as following the generic coding conventions and properly replacing the data type, Win32 APIs, and C run-time API definitions. You can see the changes in bold typeface.

        #include <tchar.h>
        // Include wchar specific header file
        TCHAR g_szTemp[MAX_STR]; // Definition of the data type as a// generic variable // Calling the generic LoadString and not W or A versions explicitly
LoadString(g_hInst, IDS_SAMPLE, g_szTemp, MAX_STR);
// Using the appropriate text macro for the title of our message box
MessageBox(NULL, g_szTemp, TEXT("This is a Unicode message box."),
MB_OK);
// Using the generic run-time version of strlen
ExtTextOut(hDC, 10, 10, ETO_CLIPPED , NULL, g_szTemp,
_tcslen(g_szTemp), NULL);

After implementing these simple steps, all that is left to do in order to create a Unicode application is to compile your code as Unicode by defining the compiling flags UNICODE and _UNICODE.

Options to Migrate to Unicode

Depending on your needs and your target operating systems, there are several options for migration from an application that is based on code pages or to one that is based on Unicode. Some of these options do have certain caveats, however.

  • Create two binaries: default compile for Windows 95, Windows 98, and Windows Me, and Unicode compile for Windows NT, Windows 2000, and Windows XP.

  • Disadvantage: Maintaining two versions of your software is messy and goes against the principle of a single, worldwide binary, introduced in Chapter 2, "Designing a World-Ready Program."

  • Always register as a non-Unicode application, converting to and from Unicode as needed.

  • Disadvantage: Since Windows does not support the creation of custom code pages, you will not be able to use scripts that are supported only through Unicode (such as those in the Indic family of languages, Armenian, and Georgian). Also, this option makes multilingual computing impossible since, when it comes to displaying, your application is always limited to the system's code page.

  • Create a pure Unicode application.

  • Disadvantage: This works only on Windows NT, Windows 2000, and Windows XP, since only limited Unicode support is provided on legacy platforms. This is the preferred approach if you are only targeting Unicode platforms.

  • Use Microsoft Layer for Unicode (MSLU). In this easy approach, you merely create a pure Unicode application, and then link the Unicows.lib file provided by the SDK platform to your project. You will also need to ship the Unicows.dll file along with your deliverables. MSLU is essentially wrapping all explicit "W" version calls made in your code, at run time, to "A"versions, if a non-Unicode platform is detected at run time. This approach is by far the best solution for migrating to Unicode and for ensuring backward-compatibility. (For more information about MSLU, see Chapter 18.)

Best Practices

When writing Unicode code, there are many points to consider, such as when to use UTF-16, when to use UTF-8, what to use for compression, and so forth. The following are recommended practices that will help ensure you choose the best method based on the circumstance at hand.

  • Choose UTF-16 as the fundamental representation of text in your application. UTF-8 should be used for application interoperability only (for example, for content sent to be displayed in browsers that do not support Unicode, or over networks and servers that do not support Unicode).
  • Avoid character-by-character processing and use the existing WCHAR system interfaces and resources wherever possible. The interaction between characters in some languages requires expert knowledge of those languages. Microsoft has developed and tested the system interfaces with most of the languages represented by Unicode-unless you are a multilingual expert, it will be difficult to reproduce this support.
  • If your application must run on Windows 95, Windows 98, or Windows Me, keep UTF-16 as your fundamental text representation and use MSLU on these operating systems. If you must support non-Unicode text, keep data internally in UTF-16 and convert to other encodings via a gateway. Use system interfaces such as MultiByteToWideChar to convert when necessary.
  • Ensure your application supports Unicode characters that require two UTF-16 code points (surrogate pairs). This should be automatic if you use existing system interfaces, but will require careful development and testing when you do not. Avoid the trap of likening surrogate pairs to the older East Asian double-byte encodings (DBCS). Instead, centralize the needed string operations in a few subroutines. These subroutines should take surrogate pairs into consideration, but should also handle combining characters and other characters that require special handling. A well-written application can confine surrogate processing to just a few such routines.
  • Don't use UTF-8 for compression-it actually expands the size of the data for most languages. If you need a real compression algorithm for Unicode, refer to the Unicode Consortium technical standard "Unicode Technical Standard #6: A Standard Compression Scheme for Unicode" available on their site, http://www.unicode.org.
  • Don't choose UTF-32 merely to avoid surrogate processing. Data size will double and the processing benefits are elusive. If you follow the earlier advice on surrogate processing, UTF-16 should be adequate.
  • Test your Unicode support with a mix of unrelated languages such as Arabic, Hindi, and Korean. For a well-written Unicode application, the system-locale setting should be irrelevant-test to verify this is the case.

You've now seen techniques and code samples for creating Win32 Unicode applications. Unicode is also extremely useful for dealing with Web content in the global workplace and market. Knowing how to handle encoding in Web pages will help bridge the gap between the plethora of languages that are in use today within Web content.


Encodings in Web Pages

Glossary
  • MLang: A Component Object Model (COM) component that provides a variety of services. These services include detecting the character encoding used by Web pages and e-mails, converting text from one encoding to another as part of an import or export operation, and displaying characters that are not included within the font specified for parts of a Web page.

All the reasons for creating Unicode Win32 applications-such as the need for universality and standardization-also apply to Web content. Creating and managing multilingual Web content and empowering users to see Web pages in different languages are the driving forces behind using a universal encoding for your Web content. There are several possibilities when it comes to encoding Web content, although some methods have disadvantages that you should be aware of.

Options for Web Encoding

Generally speaking, there are four different ways of setting the character set or the encoding of a Web page. (For more detailed information on how to set these encodings, see "Setting and Manipulating Encodings" later in this chapter.)

  • Windows code pages or ISO character encodings: With this approach, you can select from the list of supported code pages to create your Web content. The downside of this approach is that you are limited to languages that are included in the selected character set, making true multilingual Web content impossible. This limits you to a single-script Web page.
  • Number entities: Number entities can be used to represent a few symbols out of the currently selected code page or encoding. Let's say, for example, you have decided to create a Web page using the previous approach with the Latin ISO charset 8859-1. Now you also want to display some Greek characters in a mathematical equation; Greek characters, however, are not part of the Latin code page. Take, for instance, the Greek character [image], which has the Unicode code-point U+03A6. By using the decimal number entity of this code point pre-ceded by &#, the character's output will be as follows:
    •  <span> This is my text with a Greek Phi: &#934 </span>

    and the output would be:

  • This is my text with a Greek Phi:

    Unfortunately, this approach makes it impossible to compose large amounts of text and makes editing your Web content very hard.

  • UTF-16: Unlike Win32 applications where UTF-16 is by far the best approach, for Web content UTF-16 can be used safely only on Windows NT networks that have full Unicode support. Therefore, this is not a suggested encoding for Internet sites where the capabilities of the client Web browser as well the network Unicode support are not known.
  • UTF-8: This Unicode encoding is the best and safest approach for multilingual Web pages. It allows you to encode the whole repertoire of Unicode characters. Also, all versions of Internet Explorer 4 and later as well as Netscape 4 and later support this encoding, which is not restricted to network or wire capabilities. The UTF-8 encoding allows you to create multilingual Web content without having to change the encoding based on the target language. (See Figure 3-12.)
Figure 3.12 Example of a multilingual Web page encoded in UTF-8.
Figure 3.12 - Example of a multilingual Web page encoded in UTF-8.

(You can find the HTML file [Multilingual.html], corresponding to Figure 3-12, in the Samples subdirectory on the companion CD.)

Setting and Manipulating Encodings

Since Web content is currently based on Windows or other encoding schemes, you'll need to know how to set and manipulate encodings. The following describes how to do this for HTML pages, Active Server Pages (ASP), and XML pages.

HTML pages: Internet Explorer uses the character set specified for a document to determine how to translate the bytes in the document into characters on the screen or on paper. By default, Internet Explorer uses the character set specified in the HTTP content type returned by the server to determine this translation. If this parameter is not given, Internet Explorer uses the character set specified by the meta element in the document, taking into account the user's preferences if no meta element is specified. To apply a character set to an entire document, you must insert the meta element before the body element. For clarity, it should appear as the first element after the head, so that all browsers can translate the meta element before the document is parsed. The meta element applies to the document containing it. This means, for example, that a compound document (a document consisting of two or more documents in a set of frames) can use different character sets in different frames. Here is how it works:

<META HTTP-EQUIV="Content-Type" CONTENT="text/html;
charset=<value>">

You substitute <value> with any supported character-set-friendly name (for example, UTF-8) or any code-page name (for example, windows 1251). (For more information, see Appendix J, "Encoding Web Documents.")

ASP pages: Internally, ASP and the language engines it calls-such as Microsoft Visual Basic Scripting Edition (VBScript), JScript, and so forth-all communicate in Unicode strings. However, Web pages currently consist of content that can be in Windows or other character-encoding schemes besides Unicode. Therefore, when form or query-string values come in from the browser in an HTTP request, they must be converted from the character set used by the browser into Unicode for processing by the ASP script. Similarly, when output is sent back to the browser, any strings returned by scripts must be converted from Unicode back to the code page used by the client. In ASP these internal conversions are done using the default code page of the Web server. This works great if the users and the server are all using the same language or script (more precisely, if they use the same code page). However, if you have a Japanese client connecting to an English server, the code page translations just mentioned won't work because ASP will try to treat Japanese characters as English ones.

The solution is to set the code page that ASP uses to perform these inbound and outbound string translations. Two mechanisms exist to set the code page:

  • Per page, at design time: <%@CODEPAGE=<charset> %>. For example:
  • <% @ LANGUAGE=VBScript CODEPAGE=1252 %>
  • In script code, at run time: The Session.CodePage property sets the code page to use for the current sessions's string translations. In Microsoft Internet Information Services (IIS) 5.1 and later, the Response.Code-Page property defines the code page of response set to the client. Once explicitly set, the Re sponse code page overrides the Session code page, which in turn overrides the @CODEPAGE setting. For example:
  • <% @ LANGUAGE=VBScript CODEPAGE=65001 %>
<%

Response.Write (Session.CodePage)
Session.CodePage = 1252
%>

How are these code-page settings applied? First, any static content (HTML) in the .asp file is not affected at all; it is returned exactly as written. Any static strings in the script code (and in fact the script code itself) will be converted based on the CODEPAGE setting in the .asp file. Think of CODEPAGE as the way an author (or better yet, the authoring tool, which should be able to place this in the .asp file automatically) tells ASP the code page in which the .asp file was written.

Any dynamic content-such as Response.Write(x) calls, where the x is a variable-is converted using the value of Response.CodePage, which defaults to the CODEPAGE setting but can be overridden. You'll need this override, since the code page used to write the script might differ from the code page you use to send output to a particular client. For example, the author may have written the ASP page in a tool that generates text encoded in JIS, but the end user's browser might use UTF-8. With this code-page control feature, ASP now enables correct handling of code-page conversion.

The behavior of the browser set by the meta tags (described earlier) in the server-side script can be achieved by setting the Response.Charset property. Setting this property would instruct the browser how to interpret the encoding of the incoming stream. Generally this value should always match the value of the session's code page.

For example, for an ASP page that did not include the Response.Charset property, the content-type header would be

content-type:text/html

If the same .asp file included

<% Response.Charset= "ISO-LATIN-7" %>

the content-type header would be

content-type:text/html; charset=ISO-LATIN-7

XML pages: All XML processors are required to understand two transformations of the Unicode character encoding: UTF-8 (the default encoding) and UTF-16. The Microsoft XML Parser (MSXML) supports more encodings, but all text in XML documents is treated internally as the Unicode UTF-16 character encoding.

The encoding declaration identifies which encoding is used to represent the characters in the document. Although XML parsers can determine automatically if a document uses the UTF-8 or UTF-16 Unicode encoding, this declaration should be used in documents that support other encodings.

For example, the following is the encoding declaration for a document that uses the ISO 8859-1 encoding (Latin 1):

<?xml version="1.0" encoding="ISO-8859-1"?>

Just as in Win32 programming, in which there are instances where you would like to convert from one encoding to another, you can use some of the MLang functions to perform this operation in your Web content. (For more information about MLang, see Chapter 17, "MLang.")

User Override

By default, Internet Explorer uses the encoding in which a particular Web page was created or specified by the meta tag. However, a user can override the default encoding of a Web page for his or her own viewing purposes. This feature is especially helpful when the Web author does not specify an encoding or uses a wrong encoding. A user can simply right-click in a Web page and select a given encoding from the list. (See Figure 3-13 below.)

Figure 3.13 User selection of encoding for individual viewing purposes.
Figure 3.13 - User selection of encoding for individual viewing purposes.

Internet Explorer Language Support

By default, your Internet Explorer browser has the capability of supporting all languages that your Windows version supports. However, because Internet Explorer 5 and later are based on a single, worldwide binary, support for all scripts and languages are available on all versions of Internet Explorer 5 and later. This is true even when the browser is run on a Windows version that does not support all the character sets, such as when displaying Russian on a Korean version of Windows Me. A user can decide to add support for additional languages during the custom setup of Internet Explorer. Also, with its Language Encoding Auto-Select feature, Internet Explorer can usually determine the appropriate language encoding used to create a Web page.

Suppose, however, that your version of Windows does not support the detected encoding. For example, if you are running Russian Windows 98 and viewing an Arabic Web page using code page 1256, you would be prompted to download language-support components for Arabic script. The components package includes code-page information defined in files labeled c_xxx.nls, where xxx stands for the code-page number. Also included are fonts for the given script.

While Unicode offers significant benefits in terms of addressing the difficulties inherent in multilingual content on the Web, it also has an important place within the .NET Framework. Strategies for working with and manipulating various encodings in the .NET Framework follow, along with an assortment of code samples.


Encodings in the .NET Framework

Glossary
  • ASP.NET : Stands for Microsoft Active Server Pages for the .NET Framework. The new generation of Active Server Pages (ASP) files written in a managed language on the Common Language Runtime (CLR) using the .NET Framework. Also known as "ASP+" and "ASPX."

The .NET Framework is a platform for building, deploying, and running Web services and applications that provides a highly productive, standards-based, multilanguage environment for integrating existing or legacy investments with next-generation applications and services. The .NET Framework uses Unicode UTF-16 to represent characters, although in some cases it uses UTF-8 internally. The System.Text namespace provides classes that allow you to encode and decode characters, with support that includes the following encodings:

  • Unicode UTF-16 encoding. Use the UnicodeEncoding class to convert characters to and from UTF-16 encoding.
  • Unicode UTF-8 encoding. Use the UTF8Encoding class to convert characters to and from UTF-8 encoding.
  • ASCII encoding. ASCII encodes the Latin alphabet as single 7-bit characters. Because this encoding only supports character values from U+0000 through U+007F, in most cases it is inadequate for internationalized applications. You can use the ASCIIEncoding class to convert characters to and from ASCII encoding whenever you need to interoperate with legacy encodings and systems.
  • Windows/ISO Encodings. The System.Text.Encoding class provides support for a wide range of Windows/ISO encodings.

Encoding Support for Code Pages

The use of Unicode in the .NET Framework simplifies the development of world-ready applications because you no longer need to reference a code page. The .NET Framework provides support for data encoded using code pages. You can use the Encoding.GetEncoding method (Int32) to create a target encoding object for a specified code page. Specify a code page number as the Int32 parameter. The following code example creates an encoding object-enc -for the code page 1252.

[Visual Basic]
Encoding enc = Encoding.GetEncoding(1252)

[C#]
Encoding enc = Encoding.GetEncoding(1252);

After you create an encoding object that corresponds to a specified code page, you can use the object to perform other operations supported by the System.Text.Encoding class.

The one additional type of support introduced to ASP.NET is the ability to clearly distinguish between file, request, and response encodings. To set the encoding in ASP.NET for code, page directives, and configuration files, you'll need to do the following.

In code:

Response.ContentEncoding=<value>

Request.ContentEncoding=<value>

File.ContentEncoding=<value>

In page directives:

<%@Page ResponseEncoding=<value>%>
<%@Page RequestEncoding=<value>%>
<%@Page FileEncoding=<value>%>

In a configuration file:

<configuration>
   <globalization
         fileEncoding=<value>

         requestEncoding=<value>

         responseEncoding=<value>

         />
</configuration>

The following code example in C# uses the Encoding.GetEncoding method to create a target encoding object for a specified code page. The Encoding.-GetBytes method is called on the target encoding object to convert a Unicode string to its byte representation in the target encoding. The byte representations of the strings in the specified code pages are displayed.

using System;
using System.IO;
using System.Globalization;
using System.Text;

public class Encoding_UnicodeToCP
{
   public static void Main()
   {
      // Convert ASCII characters to bytes.
      // Display the string's byte representation in the
      // specified code page.
      // Code page 1252 represents Latin characters.
      PrintCPBytes("Hello, World!",1252);
      // Code page 932 represents Japanese characters.
      PrintCPBytes("Hello, World!",932);
     // Convert Japanese characters to bytes.
      PrintCPBytes("\u307b,\u308b,\u305a,\u3042,\u306d",1252);
      PrintCPBytes("\u307b,\u308b,\u305a,\u3042,\u306d",932);
   }

  public static void PrintCPBytes(string str, int codePage)
   {
Encoding targetEncoding;
      byte[] encodedChars;
// Get the encoding for the specified code page.
      targetEncoding = Encoding.GetEncoding(codePage);

// Get the byte representation of the specified string.
      encodedChars = targetEncoding.GetBytes(str);

      // Print the bytes.
      Console.WriteLine
         ("Byte representation of '{0}' in Code Page '{1}':", str,
             codePage);
      for (int i = 0; i < encodedChars.Length; i++)
         Console.WriteLine("Byte {0}: {1}", i, encodedChars[i]);
   }
}

To determine the encoding to use for response characters in an Active Server Pages for the .NET Framework (ASP.NET) application, set the value of the HttpResponse.ContentEncoding property to the value returned by the appropriate method. The following code example illustrates how to set HttpResponse.ContentEncoding.

// Explicitly set the encoding to UTF-8.
Response.ContentEncoding = Encoding.UTF8;

// Set ContentEncoding using the name of an encoding.
Response.ContentEncoding = Encoding.GetEncoding(name);

// Set ContentEncoding using a code page number.
Response.ContentEncoding = Encoding.GetEncoding(codepageNumber);

The last major area of discussion involves encodings in console or text-mode programming. In the section that follows, you'll find information on using the Win32 API and C run-time (CRT) library functions, CRT console input/output (I/O), and Win32 text-mode I/O, should you need to deal with this sort of application.


Encodings in Console or Text-Mode Programming

Programmers can use both Unicode and SBCS or DBCS encodings when programming console, or "text-mode," applications. For legacy reasons, non-Unicode console I/O functions use the console code page, which is an OEM code page by default. All other non-Unicode functions in Windows use the Windows code page. This means that strings returned by the console functions might not be processed correctly by the other functions and vice versa. For example, if FindFirstFileA returns a string that contains certain non-ASCII characters, WriteConsoleA will not display the string properly.

Always keeping track of which encoding is required by which function-and appropriately converting encodings of textual parameters-can be hard. This task was simplified with the introduction of the functions SetFileApisToOEM, SetFileApisToANSI, and a helper function AreFileApisANSI. The first two affect non-Unicode functions exported by KERNEL32.dll that accept or return a file name. As the names suggest, the SetFileApisToOEM sets those functions to accept or return file names in the OEM character set corresponding to the current system locale, and SetFileApisToANSI restores the default, Windows ANSI encoding for those names. Currently selected encoding can be queried with AreFileApisANSI.

With SetFileApisToOEM at hand, the problem with the results of WindFirstFileA (or GetCurrentDirectoryA, or any of the file-handling functions of Win32 API) that cannot be passed directly to WriteConsoleA is easily solved: after SetFileApisToOEM is called, WindFirstFileA returns text encoded in OEM, not in the Windows ANSI character set. This solution is not a universal remedy against all Windows ANSI versus OEM incompatibilities, however. Imagine you need to get text from a file-handling function, output it to console, and then process it by another function, which is not affected by SetFileApisToOEM. This absolutely realistic scenario will require the encoding to be changed. Otherwise, you will need to call SetFileApisToOEM to get data for console output, then SetFileApisToANSI and get the same text, just in another encoding, for internal processing. Another case when SetFileApisToOEM does not help is handling of the command-line parameters: when the entry point of your application is main (and not wmain), the arguments are always passed as an array of Windows ANSI strings. All this clearly complicates the life of a programmer who writes non-Unicode console applications.

To make things more complex, 8-bit code written for console has to deal with two different types of locales. To write your code, you can use either Win32 API or C run-time library functions. ANSI functions of Win32 API assume the text is encoded for the current console code page, which the system locale defines by default. The SetConsoleCP and SetConsoleOutputCP functions change the code page used in these operations. A user can call chcp or mode con cp select= commands in the command prompt; this will change the code page for the current console. Another way to set a fixed console code page is to create a console shortcut with a default code-page set (only available on East Asian localized versions of the operating system). Applications should be able to respond to a user's actions.

Locale-sensitive functions of C run-time library (CRT functions) handle text according to the settings defined by a (_w)setlocale call. If (_w)setlocale is not called in the code, CRT functions use the ANSI "C" language invariant locale for those operations, losing language-specific functionality.

The declaration of the function is:

setlocale( int category, const char *locale)

or

_wsetlocale( int category, const wchar_t *locale)

The "category" defines the locale-specific settings affected (or all of them, if LC_ALL is specified). The variable-locale -is either the explicit locale name or one of the following:

  • ".OCP"-refers to the OEM code page corresponding to the current user locale
  • ".ACP" or ""-refers to the Windows code page corresponding to the current user locale

".OCP" and ".ACP" parameters always refer to the settings of the user locale, not the system locale. Hence they should not be used to set LC_CTYPE. This category defines the rules for Unicode to 8-bit conversion and other text-handling procedures, and must follow the settings of the console, accessible with GetConsoleCP and GetConsoleOutputCP.

The best long-term solution for a console application is to use Unicode, since Unicode interfaces are defined for both the Win32 API and C run-time library. The latter programming model still requires you to set the locale explicitly, but at least you can be sure the text seen by Win32 and CRT does not require transcoding.

CRT Console I/O

When a Unicode stream I/O routine (file I/O such as fwprintf, fwscanf, fgetwc, fputwc, fgetws, or fputws, or console 'stdio' functions) operates on a file that is open in text mode (the default) or on a console, two kinds of character conversions take place:

  • Unicode-to-MBCS or MBCS-to-Unicode conversion. When a Unicode stream I/O function operates in text mode, the source or destination stream is assumed to be a sequence of multibyte characters. Therefore, the Unicode stream-input functions convert multibyte characters to wide characters (identical to calling the mbtowc function). For the same reason, the Unicode stream-output functions convert wide characters to multibyte characters (identical to calling the wctomb function).
  • Carriage return-linefeed (CR-LF) translation. This translation occurs before the MBCS-Unicode conversion (for Unicode stream input functions) and after the Unicode-MBCS conversion (for Unicode stream output functions). During input, each carriage return-linefeed combination is translated into a single linefeed character. Conversely, during output, each linefeed character is translated into a carriage return-linefeed combination.

However, when a Unicode stream I/O function operates in binary mode, the file is assumed to be Unicode, and no CR-LF translation or character conversion occurs during input or output. This makes CRT I/O complex when the application has to be globalized. Using console I/O routines of the Win32 API is a consistent and controllable way to handle multilingual data, and it should be used whenever possible.

Unicode CRT I/O functions do not provide Unicode output. Rather, they take a Unicode input and convert it to the code page defined by a prior call to setlocale, or to "C" locale (ASCII) if setlocale was not called. In the latter case, only Latin script is processed and the localized text is not displayed at all. Figure 3-14 presents the same text displayed twice, first with the Unicode Win32 API and next with the Unicode CRT. The CRT locale is not set.

Figure 3.14 CRT output example.
Figure 3.14 - CRT output example.

Win32 Text-Mode I/O

Win32 API provides two approaches for console I/O: the high-level approach using Read/WriteConsole and Read/WriteFile functions, and the low-level approach, requiring access to console screen and input buffers, keyboard, mouse, and buffer-resizing events. Only high-level functions are described here.

Both WriteConsole and WriteFile can take Unicode parameters. The difference between them is:

  • WriteConsole writes Unicode characters to console, but works on console handles only. It does not work if the output is redirected to a disk file.
  • WriteFile can take a handle of any file, so that the output can be redirected to a file or a pipe. However, the encoding is assumed to be in the current console-output code page. A Unicode input array will be displayed as garbage.

An application that displays messages that can never be redirected should stick with a WriteConsole call. Applications producing multilingual output that can be redirected can check the handle they get for output. If the output handle is of a console, use WriteConsole. Otherwise, call WriteFile with the text that matches the current console code page. In the example on the following page, the code writes Unicode output to the console, but code page-based text is sent out if the output is redirected. The assumption (not seen in the code example on the following page) is that the output string is loaded from resources that match the console-output code page.

wchar_t*   szwOut ;
DWORD      dwBytesWritten;
DWORD      fdwMode;
HANDLE     outHandle = GetStdHandle(STD_OUTPUT_HANDLE);
//...
// ThreadLocale adjustment, resource loading, etc. is skipped
//...
if( (GetFileType(outHandle) & FILE_TYPE_CHAR) &&
                           GetConsoleMode( outHandle, &fdwMode) )
{
   WriteConsoleW( outHandle, szwOut, wcslen(szwOut),
      &dwBytesWritten, 0);
}
else
{
   int nOutputCP = GetConsoleOutputCP();
   int charCount = WideCharToMultiByte(nOutputCP, 0, szwOut, -1, 0,
                      0, 0, 0);
   char* szaStr = (char*) malloc(charCount);
   WideCharToMultiByte( nOutputCP, 0, szwOut, -1, szaStr, charCount,
      0, 0);
   WriteFile(outHandle, szaStr, charCount-1, &dwBytesWritten, 0);
   free(szaStr);
}


Summary

Windows NT 3.1 was the first major operating system to support Unicode, and since then Windows NT 4, Windows 2000, and Windows XP have extended this support. Although a fully implemented Unicode application is not the only thing you need for ensuring a world-ready application, Unicode is a basic building block for other facets of globalization.

Though this chapter has presented a wide array of technical guidelines, there are several core issues to remember. First, if at all possible, it is best to use the UTF-16 coding schema to represent Unicode. However, there might be times-such as in cases involving data transmission and Web browser usage-that UTF-8 might need to be used. In addition, ensure your application handles Unicode supplementary characters by supporting surrogate pairs. Finally, if your application must run on both Unicode and non-Unicode-based Windows operating systems, MSLU helps you create a single Unicode-based application that can be run on both platforms.

By following this chapter's many other specific technical guidelines on creating Win32 Unicode applications, as well as using Unicode encodings in Web pages, in the .NET Framework, and in console or text-mode applications, you will be one step closer to creating a globalized, world-ready product.


Chapter 4: Locale and Cultural Awareness
Glossary
  • Gregorian calendar: A solar-based dating system used as the default calendar of countries in the Western hemisphere and also used widely in other parts of the world.

Implementing universally encoded applications and conforming to Unicode standards, as discussed in the previous chapter, is a major step toward creating a world-ready application. After all, it does not matter how well your application is designed to accommodate the localization process or to handle Input Method Editors (IMEs) if it cannot display the proper character in a multilingual context. Universal encoding is only one component of well-globalized software. There are still other considerations and challenges, including locale awareness.

A locale-aware program is software that observes language rules and cultural conventions that the user has selected. Representing formatted data such as the date, time, and currency symbols becomes a sensitive issue, since their final representation can have different meanings and interpretations from one language and culture to another. Take a simple example of a short date in the Gregorian calendar: 04/01/02. What date is this really?

  • April 1, 2002, if you are in the United States and the default standard is mm/dd/yy (month/day/year)
  • January 4, 2002, if you are in the United Kingdom and the default standard is dd/mm/yy (day/month/year)
  • January 2, 2004, if you are in Hungary and the default standard is yy/mm/dd (year/month/day)

(For information on how to use short-date or long-date formatting-such as April 1, 2002-see "Date and Calendar Formatting" later in this chapter.)

And yet this example only involves the Gregorian calendar, without getting into the complexity of alternative calendars such as the Hijri (Islamic lunar) or Hebrew lunar calendars. Can you imagine the confusion for a user who is look-ing, for example, at the last time his or her system has been backed up, and the possible consequences of such misinterpretations?

Microsoft Windows provides generic support for linguistic and cultural conventions through Microsoft Win32 National Language Support (NLS) application programming interfaces (APIs), as well as the Microsoft .NET Framework, which carries information on date, time, calendar, number, and currency formats. These APIs also provide sorting and character-type information for all the locales supported by the operating system. Microsoft Windows XP and Microsoft Windows .NET Server Family have integrated support for over 135 different locales, and this number will continue to grow in future releases of Windows. (See the full list at http://www.microsoft.com/globaldev/nlsweb/ and in the Samples subdirectory on the companion CD.)

A great deal of linguistic research went into creating the collection of locale information in the operating system and into the algorithms and tables used by the NLS API. You'll save yourself a lot of trips to the library, as well as the drudgery of maintaining locale information and code, by taking advantage of what the system offers. Generic code based on the NLS API will work for any locale the system supports.

It's important to mention at this early stage that because world-ready applications can never assume in which language they will display information to the user, all internal data should be stored in some predefined locale-neutral (with-out reference to language or geographic area) binary or string format. This way the data can be shared, without need for conversion, among users running your applications with different locale settings. For example, Microsoft Excel stores dates as sequential numbers known as "serial values." It also stores times as decimal fractions because time is considered a portion of a day. Thus Excel not only allows you to perform mathematical operations with dates and time, but also allows you to display the date and time in any locale format the user currently has set.

This chapter demonstrates how you can take advantage of locale-aware and culture-aware techniques to write code for Win32, Web, .NET, and console environments.

It will show you how to retrieve cultural information, and how to format dates, calendars, time, and currency. In addition, the chapter will establish how to handle sorting and stringcomparison, number formatting, as well as various international conventions involving addresses, paper size, telephone numbers, and units of measurement. By adhering to these techniques, you can avert potential misrepresentation, misinterpretation, and confusion.


Concept of Locale

To create software that is locale-aware, you'll need to first understand what is meant by the term "locale," as well as the vital role locale variables play in the development process. As you'll see, the manner in which locales are handled and interpreted differs depending on whether you're working in a traditional Win32 programming environment or within the .NET Framework. The following sections will help you understand the locale settings you'll encounter among various systems and environments.

Locale Variables

Glossary
  • Scripting engine: Component that handles character line measurement, display, caret movement, character selection, justification, shaping, and line breaking for complex scripts.
  • Multilingual User Interface (MUI) Pack: A set of language-specific resources that can be added to the English version of Windows XP Professional and the .NET Server. Once installed, MUI Packs allow the UI language of the operating system to be changed to one of 33 supported languages, depending on user preference. The MUI Pack is the same as the MultiLanguage version of Microsoft Windows 2000 Professional and Server, though it provides additional functionality.

Merriam-Webster's Collegiate Dictionary defines "locale" as "a place or locality." In Windows, this notion has been expanded to define some of the system components that deal with international support. The terminology related to locale variables has been misused in some cases, and sometimes terms differ according to whether you're dealing with Windows 2000, Windows XP, or the .NET Framework. Since you will need to work with several of the following Windows locale variables when creating locale-aware applications, it is important that you clearly understand their functionality, and their impact on the system's and the application's behavior, all from an end-user point of view first. (For more information on Windows locale variables, see the table at http://www.microsoft.com/globaldev/reference/localetable.asp.)

Language Groups and Language Collections

The different languages and scripts that an operating system can support-once the user installs them-are known as "language groups" in Windows 2000.Generally speaking, a language group contains code-page information, keyboard layouts, and fonts. Some language groups also have a scripting engine, which enables the user to edit supported scripts within the operating system.

With Windows 2000 the user can add support for many languages and scripts (including, among others, Western European, Central European, Arabic, Indic, and Turkic languages). Since Windows 2000 and Windows XP are both single, world-ready binaries, functionality for all supported scripts is available on all language versions of these operating systems. If additional languages are needed, the user only needs to install these separately either during or after setup.

In Windows XP, Microsoft has simplified the process of installing these groups by gathering languages with similar properties into a language collection. On English Windows XP, support for all European languages (including Baltic languages, Cyrillic languages, Greek, and Turkic languages)-known as the "Basic Language Collection"-is installed by default. Additional language collections (such as East Asian for Japanese, Chinese, and Korean, as well as complex scripts for Arabic, Hebrew, the Indic family of languages, and Thai) can also be installed. (After installing a language group or collection, the user will need to restart the computer.)

User scenario:

A Spanish user of English Windows XP is a translator of Telugu (from the Indic family of languages). In order to type Telugu in Notepad, for example, the user needs to install support for complex scripts. Support for additional languages cannot be installed programmatically. The user must either install it manually, or unattended setup scripts must be used.

Figure 4-1 shows a system in which support for complex scripts, for right-to-left languages, and for East Asian languages has been installed.

Figure 4-1 Regional And Language Options property sheet of Windows XP, where East Asian, complex script, and right-to-left language support have been installed.
Figure 4-1 - Regional And Language Options property sheet of Windows XP, where East Asian, complex script, and right-to-left language support have been installed.

System Locale

The system locale determines which code page-such as an Arabic 1256 or Korean 949-is used on the system by default. As explained in Chapter 3, "Unicode," a system locale's code page will be used on operating systems that use Unicode as their native encoding-such as Windows 2000 and Windows XP-to convert text data from Unicode to code page whenever dealing with legacy non-Unicode applications. In fact, in Windows XP the system locale is called "Language for non-Unicode programs." Only applications that do not use Unicode as their default character-encoding mechanism are affected by this setting; therefore, applications that are already Unicode-encoded can safely ignore the value and functionality of this setting. (See Figure 4-2.) Support for new scripts in Windows 2000 and Windows XP (such as for the Indic family, Armenian, and Georgian languages) has been provided through Unicode encoding and, therefore, these scripts are also free of any system-locale limitations.

Figure 4-2 System locale can be set from the Advanced tab of the Regional And Language Options property sheet in Windows XP.
Figure 4-2 - System locale can be set from the Advanced tab of the Regional And Language Options property sheet in Windows XP.

In this example, the system locale has been set to Romanian (Windows code page 1250 and Original Equipment Manufacturer [OEM] code page 8521), a Central European language. This means that all Central European-language applications that are based on code pages can run safely in this configuration, since they are part of the same language collection.

So, for example, a Polish application (another Central European language) does not need a system-locale change. Also, since English is part of the invariant American Standard Code for Information Interchange (ASCII) range of all code pages, English legacy applications will always run properly.

Sometimes there is no noticeable difference between two system locales. For example, the German (Standard) and German (Austria) system locales are identical, since they share the same OEM and Windows code pages; therefore, the behavior of non-Unicode-based applications will be identical in both scenarios. In general, system locales of a language group are very similar and might only be different in the OEM code page.

As its name suggests, the system locale is a unique setting for each system. Only an administrator has the right to change the system locale, and the computer will need to be restarted in order for changes to take effect. The administrator can only select a system locale if the appropriate language group-and its associated script support-is installed.

The system locale is a system variable that cannot be changed programmatically. The only way to change it is for an administrator to do so manually.

User scenarios:

  • A user running German Windows XP wants to run a Japanese application that was designed for Japanese Microsoft Windows 95. The user has to select Japanese as the system locale to do this. German legacy applications will not run flawlessly anymore; for example, German umlauts will not be displayed correctly.
  • The same German user wants to type Japanese text in a German legacy application and must select Japanese as a system locale to do so. This will limit the user's ability to enter German text in legacy applications; for example, German umlauts will not be displayed correctly.
  • An Arabic user wants to type Arabic, French, and English in an Arabic legacy application. The user has to select Arabic as a system locale. The Arabic Windows code page contains most of the characters used for French and all of the English characters.

User Locale

The user locale determines which default settings a user wants for formatting dates, times, currency, and large numbers. Although it's presented as a language (some in a combination with a country), it's not a language setting. That is, choosing the Hebrew user locale means that the user wants to adhere to the standards of Israel, not really of the Hebrew language. To avoid any confusion with this naming, the .NET Framework calls the user locale "culture information."

As its name implies, the user locale (known in Windows XP as "Standards and formats") is a variable that each individual user can set. This can be done on the fly by selecting changes from the Regional Options tab in the Regional And Language Options property sheet. (See Figure 4-3.) Locale-aware applications should use this value to display formatted data.

Figure 4-3 User locale set to Konkani.
Figure 4-3 - User locale set to Konkani.

When changes are made, all locale-aware applications should monitor the window message WM_SETTINGCHANGE and should be able to update their displayed data accordingly. Numbers, currency, date, and time are some of the variables that are affected by the user-locale setting. The user locale is a user variable that cannot be changed programmatically. The only way to change it is for the user to do so manually.

User scenario:

An American user living in Monaco and running an English version of Windows XP sets the user locale to French (Monaco) in order to use the local standards for date, time, and currency.

Input Locale

Known as "input locale" in Windows 2000 and "input language" in Windows XP, this variable describes a language a user wants to enter into an application (not necessarily type) and the method of input. There can be multiple input locales installed and the user can switch between them. The default input locale is the locale that is active when a new application is started (or in some applications, when a new window is opened). Switching to a different input locale is done on a per-thread basis; that is, you can have two different input locales in two different applications.

User scenarios:

  • The user has elected to type Italian with the Italian keyboard layout. This same user also wants to provide German input by using a voice-recognition engine, and Chinese by using an IME. Additional input locales can be installed and used on the fly by selecting changes from the Languages tab within the Regional And Language Options property sheet, provided that the language support for the target language has already been installed. (See Figure 4-4.)
  • Figure 4-4 Each user can add and remove input languages from the Languages tab of the Regional And Language Options property sheet.
    Figure 4-4 - Each user can add and remove input languages from the Languages tab of the Regional And Language Options property sheet.
  • A secretary wants to provide input for the names of international companies in each company's native language. This is possible by installing desired input locales and toggling between them. (See Figure 4-5.)
  • Figure 4-5 List of installed input locales. The user can toggle between them by using right Alt+Shift or by selecting one of them from the taskbar.
    Figure 4-5 - List of installed input locales. The user can toggle between them by using right Alt+Shift or by selecting one of them from the taskbar.

Location or Geographic ID

This variable is available in Windows XP (and was also available in Microsoft Windows Millennium Edition [Windows Me] though not in Windows 2000) to define the country or location where the user lives. Each user can change this variable on the fly by selecting changes from the Regional Options tab of the Regional And Language Options property sheet. Any changes made are also applied on the fly. By selecting a particular location, the user has set a variable that a Web service (such as one that deals with weather) can check, thus allowing the Web service to deliver information and services specific to the region or country the user has selected.

User scenario:

A user traveling to Malaysia wants to get the weather forecast for that locale and sets the location to that country. (See Figure 4-6 on the following page.) The location is a user variable that cannot be changed programmatically. The only way to change it is for the user to do so manually.

Figure 4-6 Regional Options with the location variable set to Malaysia.
Figure 4-6 - Regional Options with the location variable set to Malaysia.

Thread Locale

The thread locale defaults to the currently selected user locale and determines the formatting of dates, times, currency, and large numbers for the thread. It can be changed programmatically using the API SetThreadLocale, but in most cases the thread locale should not be overwritten.

On Microsoft Windows NT 4, a lot of applications used the thread locale to define which language resources should be retrieved and displayed. However, this practice represents a misusage of the thread locale. (In Windows 2000 and Windows XP, the system's resource loader does not default to the thread locale variable.) As you'll see, resource languages should always be driven by and follow the user interface (UI) language variable.

User scenario:

A financial stock-trading application for the New York Stock Exchange used in a bank worldwide has to display the time, date, and stock prices in U.S. formats. This application uses SetThreadLocale to set the thread locale to English (United States) and can use the NLS APIs to format dates, times, and stock prices.

UI Language

This variable allows each user to select the language of the UI for such things as dialog boxes, menus, and Help files. This option is only available on the MUI Pack of Windows XP Professional and on the MultiLanguage version of Windows 2000 Professional. (For more information on MUI, see Chapter 6, "Multilingual User Interface [MUI].")

It's important to distinguish between the system UI language and the user UI language. Though it is true that the user UI language is sometimes the same as the system UI, in other instances it is not. The system language is the language of the localized version that was used to set up Windows 2000 Professional or Windows XP Professional. All menus, dialog boxes, error messages, and Help files are in this language, except on multilanguage versions (such as on the MUI Pack of Windows XP Professional and the MultiLanguage version of Windows 2000 Professional), where the user can select a different language.

The user UI language on a non-MUI machine would be the same as the system UI language. With MUI, however, the user can change the language by clicking the Languages tab within the Regional And Language Options property sheet. (See Figure 4-7.) To see the effect of this change, the user will have to log off and then log back on.

Figure 4-7 The UI language option is only available on MUI systems (and on the MultiLanguage version of Windows 2000 Professional). In this case the UI language is being set to Hebrew (though changes have not yet been applied).
Figure 4-7 - The UI language option is only available on MUI systems (and on the MultiLanguage version of Windows 2000 Professional). In this case the UI language is being set to Hebrew (though changes have not yet been applied).

MUI is only available on Microsoft Windows 2000 Professional, Microsoft Windows 2000 Server Family, Microsoft Windows XP Professional, and Microsoft Windows .NET Server Family. The UI language is a user variable that cannot be changed programmatically. The only way to change it is for the user to do so manually.

Browser Language Setting

Microsoft Internet Explorer versions 4 and later share the same functionality across all Windows platforms. Since versions of Windows prior to Windows 2000 did not offer the same flexibility in terms of locale selection (no distinction between user and system locale, for example), Internet Explorer allows the user to select the browser language setting. Web sites can use this setting to offer their content in the user's selected language and to format and display data using the selected locale standards, as in the case of http://www.msn.com. To try out the browser language setting, from Internet Options, click the Languages button (see Figure 4-8), set your browser language to French (France), for example, and go to the MSN Hotmail site at http://www.hotmail.com. MSN Hotmail reads the selected browser setting and redirects you to the French Hotmail version.

All the locale variables described previously function independently of one another, and changing one of them does not affect the setting of the other variables. To summarize this section, here's an example involving an English version of Windows XP.

Figure 4-8 The browser language setting can be accessed through Internet Options on the Tools menu. Click the Languages button.
Figure 4-8 - The browser language setting can be accessed through Internet Options on the Tools menu. Click the Languages button.

User scenario:

Michelle from Chile is living in the United States. Her location is set to the United States because she wants to use a national Internet Service Provider (ISP) and also wants to get the weather forecast for the United States. Her user locale is set to Spanish-Chile because she wants to see formatted data within her native country's standards.

Michelle is also fluent in Korean. She recently purchased a Korean word-processor (a non-Unicode application) that she wants to run on her English machine. She sets her system locale to Korean. To start using her application, in addition to using the Englishkeyboard, she installs a Korean IME as a second input locale. Her husband, sharing the same computer and not being comfortable with English, can set the UI language to Spanish for his own account (if running an MUI version)!

Now that you have seen the basic functionality of locale variables, and their impact on the system's and an application's behavior, you will be better equipped to apply the technical concepts discussed in the remainder of the chapter. The section that follows will help you understand how to use locale variables when dealing with Win32 applications. It will also discuss working with NLS APIs, language IDs, and sort IDs.

NLS and Locale ID

Glossary
  • Bopomofo: A set of characters used to teach the phonetics of Chinese. These characters are used in teaching materials such as dictionaries, but do not appear in the actual writing of the Chinese language.
  • Korean KSC: Character encoding established by the Korean Industrial Standards Association. Now known as "KSX."

To provide you with a general comprehension of the settings you'll encounter among various systems and environments, the previous section discussed all locale variables. However, among all the locale variables described, only the user locale, the location, the browser language setting, and the thread locale deal with locale awareness in applications. These particular variables will be the focus of the remainder of this chapter.

As mentioned previously, NLS APIs are used in the traditional Win32 programming paradigm in order to provide culture-sensitive or locale-sensitive information. NLS is an unfortunate misnaming, since the settings it deals with are not always "national," but instead can be cultural or even custom-defined. NLS APIs provide the right tools to eliminate any locale-specific assumptions from your application, allowing you to represent data in the formatting that corresponds to the currently selected user locale. For example, suppose that you want torepresent the system time; the English-centric way of doing it would be something like:

SYSTEMTIME SystemTime;
GetSystemTime(&SystemTime);
stprintf(g_szTemp, TEXT("today is: %d/%d/%d"), SystemTime.wMonth,
         SystemTime.wDay, SystemTime.wYear);

Without taking into account the type of date and calendar format that the user is expecting, the English (United States) Gregorian date format of mm/dd/yyyy is imposed here. To overcome this problem, NLS APIs provide three sets of APIs that allow you to:

  • Retrieve a given locale value.
  • Query information about a given locale.
  • Format data for a given locale (in this case, to represent the system time).

The modular approach would be to query the selected user locale, get the default calendar format for that retrieved locale, and format the system time using the default calendar format. A common argument to use for almost all of the NLS APIs is a locale ID, or LCID. An LCID is a combination of a language ID (LANGID) and a sort ID. Here is how it works.

Language ID

A language ID is a WORD value composed of a primary language combined with a sublanguage ID that defines the country, region, or even a writing system where this language is used. For example, English is a primary language and Australia is a sublanguage, yielding the English (Australia) language ID. (For the list of system-defined LANGIDs, see the Microsoft Windows Platform SDK at http://msdn.microsoft.com, and Appendix D, "Table of Language Identifiers.")

For most of the locales, the LANGID is sufficient to clearly identify cultural characteristics of a given language. However, a few locales have an additional parameter that would make them distinguishable from one variation to another; that parameter is the sort order, or the way characters (or ideographs) are sorted. For example, the German (Germany) language has two types of sorting: Dictionary and Phone Book (DIN) that treat umlaut characters differently. To identify these two versions of sort order for German (Germany), the notion of LANGID has been expanded with a sort ID to create a DWORD value called the "LCID." (See Figure 4-9.)

Figure 4-9 LANGIDs and Locale IDs.
Figure 4-9 - LANGIDs and Locale IDs.

As mentioned, for most of the locales a sort ID is unnecessary. Therefore, the SORT_DEFAULT value is set in their respective LCIDs. (For a list of Windows-supported sort methods, see Appendix F, "Sorting Methods for Select Languages.")

The following section will discuss how to work with locale-specific information within the .NET Framework. As you'll see, culture codes, subculture codes, and the CultureInfo class all play an important role within this framework.

The CultureInfo Class in the .NET Framework

Glossary
  • Neutral culture: In the .NET Framework, refers to cultures identified by language only (with no associated geographic region). A neutral culture is indicated by a two-letter code, such as "de" for German.

The .NET Framework took a new approach to the definition of locale identification and readjusted the misnaming introduced by NLS APIs with its introduction of culture codes (associated with a language) and subculture codes (associated with a country or region). The CultureInfo class from the System.Globalization namespace contains culture-specificinformation, such as the language, country or region, calendar, and other cultural conventions. This class also provides the information required for performing culture-specific operations, such as casing, formatting dates and numbers, and comparing strings.

The CultureInfo class specifies a unique name for each culture based on the Request for Comments (RFC) 1766 from the Internet Engineering Task Force (IETF). (For more information, go to http://www.ietf.org.) The name is a combination of a two-letter lowercase culture code associated with a language (known as a "primary language ID" in traditional NLS) and, if required, a two-letter uppercase subculture code (known as a "sublanguage" in traditional NLS) associated with a country, region, or even a writing system. The subculture code follows the culture code, separated by a dash (-). Examples include "ja-JP" for Japanese in Japan, "en-US" for U.S. English, or "de-DE" for German in Germany (as opposed to an alternate such as "de-AT" for German in Austria). In this context, a neutral culture is specified by only the two-letter lowercase culture code. For example, "fr" specifies the neutral culture for French, and "de" specifies the neutral culture for German. A specific culture is identified by the culture code followed by the two-letter uppercase subculture code. For example, "fr-FR" specifies French in France and "fr-CA" specifies French in Canada.

Some culture names have suffixes that specify the sort order or the script; for example, "-Cyrl" specifies the Cyrillic script, "-Latn" specifies the Latin script. Thus Azeri (Cyrillic)-Azerbaijan would be indicated by "az-AZ-Cyrl." The culture identifier is the hexadecimal value associated with these culture names and is equivalent to NLS LCIDs. (For a complete list of culture names and identifiers, see Appendix K, "CultureInfo Names and Identifiers in the Microsoft .NET Framework Library.")

The RegionInfo class, also from the System.Globalization namespace, contains information about a given country or region. In contrast with CultureInfo, RegionInfo does not represent user preferences and does not depend on the user's language or culture. A good example of this category of information is the RegionInfo.IsMetric property of RegionInfo that retrieves a Boolean value, indicating whether the country or region uses the metric system for measurements. Another example is the RegionInfo.CurrencySymbol property, which returns the default currency symbol of a given region or country.

The RegionInfo.Name property is one of the two-letter codes defined in the International Organization for Standardization (ISO) 3166 for representing a particular country or region. (For a complete list of culture names and identifiers, see Appendix L, "RegionInfo Names and Identifiers in the Microsoft .NET Framework Library.")

The sections that follow will show you the specific tasks involved in writing an application that is locale-aware. All code samples in the rest of this chapter are given in the following programming languages:

  • C for Win32 programming
  • Microsoft Visual Basic Scripting Edition (VBScript) for Web programming
  • Microsoft C# for the .NET Framework


Retrieving the Right Locale

Since all APIs and functions that allow locale-specific formatting take as an argument the locale ID or the culture name for which this formatting should be performed, the first step in writing locale-aware software is to retrieve the proper locale. How you retrieve this locale will depend on whether you're dealing with Win32 applications, Web pages, or the .NET Framework.

Retrieving the User Locale in Win32

The user locale defines a user's preferences for formatting locale-sensitive data (such as date, time, and currency). Your application should use the appropriate user locale to display formatted data.

Use the GetUserDefaultLCID API to retrieve the user locale value shown as follows:

LCID uiVal;
uiVal = GetUserDefaultLCID();

Since this is a user-specific setting, there is no API made public to alter its value. The user must manually change the value. Besides retrieving the currently selected user locale, you might also want to enumerate and find out about all installed or supported locales in the system. An installed locale is a locale for which the appropriate code page and language support have been installed on the system; a supported locale is the locale for which the system provides appropriate NLS information with appropriate script support installed. The code sample that follows enumerates all installed locales in the system and finds the appropriate language name associated with each LCID.

EnumSystemLocales(EnumLocalesProc, LCID_INSTALLED);

// the enumeration callback function
BOOL CALLBACK EnumLocalesProc(LPTSTR lpLocaleString)
{
LCID uiCurLocale;
TCHAR szCurNam[STR_LEN];

 if (!lpLocaleString)
      return FALSE;

// This enumeration returns the LCID as a character string while
//    we will be using numbers in other NLS calls.
   uiCurLocale = uiConvertStrToInt(lpLocaleString);

// Get the language name associated with this locale ID.
   GetLocaleInfo(uiCurLocale, LOCALE_SLANGUAGE, szCurName, STR_LEN);
   return TRUE;
}

The big limitation of this enumeration is the fact that LCIDs are returned as character strings, and yet in other NLS APIs, a numeric value is expected. To address this, you can write your own string-to-integer transformation function. (Keep in mind that the character string is the actual hexadecimal representation of the LCID.) In the code sample you've just seen, the call to uiConvertStrToInt is an internal call to this transformation function.

Another thing that's noteworthy about this code sample is the call to a commonly used API called GetLocaleInfo. GetLocaleInfo can retrieve a variety of locale-specific information, from localized names for days of the week to the default paper size used for each locale. You can either specify the LCID of the locale for which the information is being retrieved, or use the predefined flag LOCALE_USER_DEFAULT. The latter defaults to the currently selected user locale and saves you from having to call GetUserDefaultLCID.

The user locale represents the user's preference for formatting locale- sensitive data for Win32 applications. In Web content, you will need to retrieve the browser language setting to represent content that corresponds to the language and locale of the user.

Retrieving the Browser Language Setting

By default, the global locale of your Web content will always match the following:

  • The user locale of the hosting server for HTML or Dynamic HTML (DHTML) pages
  • The system locale of the hosting server for Active Server Pages (ASP)

When it comes to locale-aware and culture-aware Web design, it's important to represent the data in the client-side format rather than defaulting to the server-side setting. Suppose your server is hosted on an English machine with an English (United States) user locale, but its content is viewed by an English (United Kingdom) user, where the date formatting goes from the English (United States) format of mm/dd/yy to the English (United Kingdom) format of dd/mm/yy.

The browser language setting is commonly used by multilingual Web sites to define the default language in which their content should be represented to the user, as well as the locale in which the data formatting should follow its standards. The technique of trying to get the browser-setting information from the client side is usually referred to as "browser sniffing." The user can set the language in the browser. For example, in Internet Explorer choose Internet Optionsfrom the Tools menu, and then click the Languages button of the General tab to choose one or more preferred languages. Other browsers support the same functionality; Netscape 4.x and 6.x allow the user to set this information by clicking Preferences in the Edit menu. If using Internet Explorer, the user can also change regional settings and have the browser automatically pick up the new language choice. This information is sent to the server in the form of a server variable known as "HTTP_ACCEPT_LANGUAGE." You can retrieve it in ASP with VBScript code such as the following:

Dim stLang
stLang = Request.ServerVariables("HTTP_ACCEPT_LANGUAGE")

This string, now sitting in the stLang variable, can be used in many different ways to control the content of your site. For example, you can:

  • Use the Microsoft Internet Information Services (IIS) 5 Server.Transfer or Server.Execute methods to move to a localized page.
  • Use the VBScript 5 SetLocale function to change the locale of the page, which will affect the display of the date, time, number, percent, and currency formats.
  • Query a database for content (assuming some or all of your site is database-driven).
  • Provide localized content through components.
  • Provide localized content directly in your ASP code.

(For more information on the possibilities just listed, see Chapter 6.)

The only difficulty in retrieving the browser language is that if you select multiple languages, your HTTP_ACCEPT_LANGUAGE string will look something like the following:

et,en-us;q=0.8,fa;q=0.6,it;q=0.4,fr;q=0.2

In this string, locales are separated by a comma-in this case, Estonian, English (United States), Farsi, Italian (Italy), and French (France). The "q=" represents the priority of each language to help create a fallback mechanism. Suppose the preferred language is Estonian (q=1 by default). However, if you are not offering any support for this language, you can parse the HTTP_ACCEPT_ LANGUAGE string for the next preferred language (q=0.8 for English).

You can then use the SetLocale function to set the global locale of your Web content. (This setting will only be applicable to your session and context.) The following example explicitly sets this locale to Estonian, and the original variable keeps the previous or original locale:

Dim original
    original = SetLocale("et-EE"))

The navigator object navigator.userLanguage can also be used to retrieve the browser locale. (For an ASP sample application that handles browser sniffing and that represents formatted data in browser-locale standards, see the Samples subdirectory on the companion CD.)

While the browser language setting is important within Web content, in the .NET Framework the CultureInfo class is what you'll need to retrieve. In this environment, the particular culture that is associated with both a language and a region is used to determine the appropriate formatting conventions.

Retrieving the Current Cultureinfo

As mentioned previously, the CultureInfo class holds culture-specific information, such as the associated language, country or region, calendar, and cultural conventions. The CurrentCulture property gets the CultureInfo instance that represents the culture used by the current thread and returns the value of Thread.CurrentCulture. By default, the CurrentCulture property is set to the currently selected user locale of the system, as set by the user in the Regional And Language Options property sheet. Properties and standards of this default culture should be used to represent formatted data to the user. You can also explicitly set the value of CurrentCulture to a given culture name in your code (for the purpose and usage of your application only and not through the system). The following example sets the CurrentCulture to Finnish (Finland) in C#:

Thread.CurrentThread.CurrentCulture = new CultureInfo("fi-FI");

The CurrentCulture property expects a culture that is associated with both a language and a region, such as ("es-ES") for Spanish in Spain. Because a language is often spoken in more than one country or region, the regional information is necessary to determine the appropriate formatting conventions to use. A neutral culture cannot be used for the creation of the CurrentCulture property. If you only have access to a neutral culture, you can create a CultureInfo object in the format that the CurrentCulture property expects. This is done by using the CultureInfo.CreateSpecificCulture method. This method maps a neutral culture to the default specific culture it is associated with, and then creates a CultureInfo object that represents that specific culture. The following code example uses the CultureInfo.CreateSpecificCulture method to map the neutral culture "it" for Italian to the specific culture "it-IT." It then creates a CultureInfo object for "it-IT" and uses it to initialize the value of the CurrentCulture property.

Thread.CurrentThread.CurrentCulture =
   CultureInfo.CreateSpecificCulture("it");

Similar to the Win32 NLS paradigm, the CultureInfo class allows you to enumerate all installed or supported cultures. The CultureInfo.GetCultures method can be used for that purpose, as shown in the following example where all supported locales are enumerated:

using System;
using System.Globalization;

public class printClass
{public static void Main()
   {
      foreach (CultureInfo ci in
      CultureInfo.GetCultures(CultureTypes.AllCultures))
      {

         Console.WriteLine(ci);
      }
   }
}

The .NET Framework allows you to get an LCID from a Culture object by using the LCID property of CultureInfo.

In addition to retrieving the culture, another task involved in creating an application that's locale-aware is to properly format dates and calendars in accordance with the locale. You'll need to account for the many differences that exist among various countries and regions.


Date and Calendar Formatting

Glossary
  • Japanese Emperor Era: This Japanese calendar works exactly like the Gregorian calendar, except that the year and era are different. The Japanese calendar recognizes one era for every emperor's reign. The current era is the Heisei era, which began in the Gregorian calendar year 1989. The era name is typically displayed before the year. For example, the Gregorian calendar year 2002 would be Heisei 14 in the Japanese calendar.
  • Taiwan calendar: The Taiwan calendar works exactly like the Gregorian calendar, except the year is different. Years in Taiwan are calculated from the year 1912, when Dr. Sun Yat-sen founded a republic in China. 1912 represents Year 1 of the Taiwan calendar. To calculate the year in the Taiwan calendar, take the year in the Gregorian calendar and subtract 1911. For instance, the Gregorian calendar year 1971 would be 60 in the Taiwan calendar.
  • Hijri (Islamic lunar): Since the Islamic calendar is purely lunar, as opposed to solar or luni-solar, the Islamic (Hijri) year is shorter than the Gregorian year by about 11 days, and months in the Islamic (Hijri) year are not related to seasons, which are fundamentally determined by the solar cycle.
  • Hebrew lunar: A calendar based on the cycle of the moon around the Earth. The length of this cycle, the lunar month, is about 29½ days. Twelve lunar months make, therefore, about 354 days.
  • Korean Tangun Era: According to Korean legend, the god-king Tangun founded the Korean nation in BC 2333. Early Korea used a lunar calendar. As the rest of the world encroached on Korea, it eventually went to the solar, Gregorian calendar. Yet much of the country still uses the lunar calendar to keep track of births and deaths and some traditional holidays.
  • Thai calendar: The Thai calendar uses the Buddhist Era BE, which is 543 years older than the Christian Era AD. To convert from a BE date to an AD date, subtract 543. Thus BE 2543=2000. Before April 1, AD 1889, Thailand used a lunar calendar of 12 or 13 months each with 29 or 30 days, each month starting with the new moon. The Gregorian calendar was adopted on April 1, 1889.

Date formatting is not constant throughout the world. Although each date basically displays the day, month, and year, their presentation order and separators vary greatly. In fact, there might be many differences between regions within the same country. To help illustrate this, take a look at two basic date formats for English (United States):

Long Date (Tuesday, October 12, 1954)

Short Date (10/12/54)

Now compare these formats for English (United States), Spanish (Mexico), and Japanese:

Long Date

English (United States): Tuesday, October 12, 1954

Spanish (Mexico): martes, 12 de octubre de 1954

Japanese: 1954 [image] 10 [image] 12

Obviously the names of the months and days of the week are different from locale to locale, but in Spanish (Mexico), the day comes before the month, everything is lowercase, and the preposition "de" (Spanish for "of") has been added. In Japanese, the day of the week is not displayed, and the translations for day, month, and year act more like separators.

Short Date

English (United States): 10/12/54

Spanish (Mexico): 12/10/54

Japanese: 54/10/12

In the short date, in Spanish (Mexico) the order is once again day/month/year as compared to English (United States) where it is month/day/year. In Japan, the order is year/month/day. This can cause some real confusion if not watched carefully.

All the examples given so far were based on the Gregorian calendar. However, to add to the complexity, world-ready products should also take into consideration other calendaring systems in use worldwide. For example, there are the Japanese, the Buddhist era, the Hijri, the Hebrew lunar, and the Taiwan calendars. One of the major differences between calendars is that each calendar could have a different year value. For example, the Gregorian year 2000 is the twelfth year in the Japanese Heisei era and the year 1421 in the Hijri calendar. The first day of the year might not start on January 1. The Chinese New Year was on February 5 of the Gregorian year 2000. The length of the year and months might also vary, as well as ways of handling leap years. Or even within the same calendar, the first day of the week might start on another day besides Sunday, depending on the culture. For instance, in most of the European countries that use the Gregorian calendar, the start of the week is Monday. Unlike English (United States), there are other locales that use more than one calendar type, such as Korean. (See Figure 4-10.)

Figure 4-10 Available calendar types for the Korean locale.
Figure 4-10 - Available calendar types for the Korean locale.

The Regional And Language Options property sheet allows the user to:

  • Select an alternative calendar (if applicable to the selected locale).
  • Define a two-digit year range for each one of the available calendars.
  • Define a default long-date and short-date formatting for each available calendar type.

Although not all international calendars are provided for, there are many supported calendar types on Windows 2000 and Windows XP. (See Table 4-1.)

Table 4-1 Supported calendar types in Windows 2000 and Windows XP.

Value | Meaning
ENUM_ALL_CALENDARS | All applicable calendars for the specified locale
CAL_GREGORIAN | Gregorian (localized)
CAL_GREGORIAN_US | Gregorian (English strings always)
CAL_JAPAN | Japanese Emperor Era
CAL_TAIWAN | Taiwan
CAL_KOREA | Korean Tangun Era
CAL_HIJRI | Hijri (Islamic lunar)
CAL_THAI | Thai
CAL_HEBREW | Hebrew (lunar)
CAL_GREGORIAN_ME_FRENCH | Gregorian Middle East French
CAL_GREGORIAN_ARABIC | Gregorian Arabic
CAL_GREGORIAN_XLIT_ENGLISH | Gregorian transliterated English
CAL_GREGORIAN_XLIT_FRENCH | Gregorian transliterated French

The interesting fact about NLS APIs and .NET support is that you don't need to know the target languages or how the names of days and months are translated into the target language. The system returns the translated values to you automatically.

You have seen some of the variations that exist among calendars throughout the world. The following sections provide solutions for handling the motley assortment of formats.

Manipulating Calendars and Formatting Dates in Win32

The easiest and most efficient method of formatting the date in a way that's locale-aware is to use the GetDateFormat API. This API allows the formatting of any given date in any supported locale format, according to the currently selected default calendar type and date format. Here is a code sample that shows how this API works:

TCHAR g_szBuf1[MAX_STR];
GetDateFormat(
  LOCALE_USER_DEFAULT,  // the locale for which the formatting is
                        //    being done
  DATE_LONGDATE,        // date format (long, short, ...)
  NULL,                 // the date to be formatted (here current
                        //    system date)
  NULL,                 // style of date format
  g_szBuf1,             // output buffer
  MAX_STR               // size of output buffer
);
MessageBox(NULL, g_szBuf1, TEXT("Long date format"), MB_OK);

The result of this code's execution for English (United States) and English (Canada) is illustrated in Figure 4-11.

Figure 4-11 Long date formatted for English (United States) and English (Canada).
Figure 4-11 - Long date formatted for English (United States) and English (Canada).

In more advanced date processing and display, you can use the lpFormat parameter of GetDateFormat to represent the date in any alternative formats that the current locale supports. The elements in Table 4-2 can be used to construct a format-picture string. If you use spaces to separate the elements in the format string, these spaces will appear in the same location in the output string. The letters must adhere to the casing conventions shown in Table 4-2. (For example, the correct formatting for "month as digits with leading zero for single-digit months" would be "MM" not "mm.") Characters in the format-picture string that are enclosed in single quotation marks will appear in the same location and will be unchanged in the output string.

Table 4-2 Date and time picture string elements.
Picture | Meaning
d | Day of the month as digits with no leading zero for single-digit days.
dd | Day of the month as digits with leading zero for single-digit days.
ddd | Day of the week as an abbreviation. The function uses the LOCALE_SABBREVDAYNAME value associated with the specified locale.
dddd | Day of the week as its full name. The function uses the LOCALE_SDAYNAME value associated with the specified locale.
M | Month as digits with no leading zero for single-digit months.
MM | Month as digits with leading zero for single-digit months.
MMM | Month as an abbreviation. The function uses the LOCALE_SABBREVMONTHNAME value associated with the specified locale.
MMMM | Month as its full name. The function uses the LOCALE_SMONTHNAME value associated with the specified locale.
y | Year as last two digits, but with no leading zero for years less than 10.
yy | Year as last two digits, but with leading zero for years less than 10.
yyyy | Year represented by full four digits.
gg | Period/era string. The function uses the CAL_SERASTRING value associated with the specified locale. This element is ignored if the date to be formatted does not have an associated era or period string.

For example, to get the date string:

"Wed, Aug 31 94"

use the following picture string:

"ddd',' MMM dd yy"

You can also retrieve all standard date formats available for a given locale by using the EnumDateFormatsEx API. Here is how it works:

// Enumerate all long date formats for the current user locale.
EnumDateFormatsEx(EnumDateFormatsProc, // callback function for
                                       //    enumeration
   LOCALE_USER_DEFAULT,                // flag for the user locale
   DATE_LONGDATE);                     // date formats

// the callback function and what it contains...
BOOL CALLBACK EnumDateFormatsProc(LPTSTR lpDateFormatString,
      CALID CalId)
{

   // If enumeration completed, stop...
   if (!lpDateFormatString)
      return FALSE;

   // Format today's date for the current locale using
   // the retrieved format.

GetDateFormat(LOCALE_USER_DEFAULT, NULL, NULL, lpDateFormatString,
      g_szBuf1, MAX_STR);

   return TRUE;

}

Execution of the previous code would give the following result in English (United States) and English (Canada) user locales, respectively. (See Figure 4-12.)

Figure 4-12 Available long-date formats for English (United States) and English (Canada).
Figure 4-12 - Available long-date formats for English (United States) and English (Canada).

As you probably noticed in the last code sample, one of the arguments of the enumeration callback function for date formats is the calendar type (CALID). As mentioned earlier, for some locales more than one calendar type is available. For example, if the user locale is set to Hebrew, then the enumeration would return long-date formats for both Gregorian and Hebrew lunar calendars. For most of the applications, the basic functionality made available by GetDateFormat, which allows the date to be represented in the currently selected calendar and format, is enough. However, some applications might want to have more control over the way this information is presented and eventually select the calendar type in which the date is being displayed. There are several Win32 NLS APIs that allow you to manipulate calendar types.

The EnumCalendarInfoEx API can be used to enumerate all calendar information (such as names of calendars, names of days of the week, and names of months) for all applicable and available calendar types pertaining to a given locale. The code sample enumerates the native names of all supported calendars.

// Enumerate the native calendar names for all available calendar
// types that correspond to the current user locale.
EnumCalendarInfoEx(EnumCalendarInfoProc,  // enumeration callback
                                          // fuction
   LOCALE_USER_DEFAULT,  // locale - any valid LCID
   ENUM_ALL_CALENDARS,   // does enumeration for all supported
                         //    calendars
   CAL_SCALNAME);        // calendar info (return the calendar name)

// The callback function will look like:
BOOL CALLBACK EnumCalendarInfoProc(LPTSTR lpCalendarInfoString,
   CALID Calendar)
{
   if (!lpCalendarInfoString)
      return FALSE;
   MessageBox(NULL, g_szBuf2, TEXT("Calendars names"), MB_OK);
   return TRUE;
}

Execution of the previous code sample would give the following result on English (United States) and Arabic (Tunisia) user locales, respectively. (See Figure 4-13.)

Figure 4-13 Available calendar types for English (United States) and Arabic (Tunisia).
Figure 4-13 - Available calendar types for English (United States) and Arabic (Tunisia).

The GetLocaleInfo API with the LOCALE_ICALENDARTYPE flag also allows you to retrieve the default calendar type currently selected by the user. Once you have retrieved the calendar you want, you can use GetCalendarInfo to retrieve specific information about that particular calendar. This information includes the following:

  • Long, short, and year/month date formats
  • Abbreviations and full names of months
  • Abbreviations and full names of days of the week
  • An integer value indicating the upper boundary of the two-digit year range
  • Native name of the calendar

In a nutshell, Win32 NLS APIs give your application full control over the way a date can be represented by letting you select a given calendar type and a given formatting type for that calendar. In fact, NLS APIs give you more flexibility than you would ever need. As stated earlier, GetDateFormat is probably enough to format dates in a way that's locale-aware.

Date Formatting in Web Pages

In "Retrieving the Browser Language Setting" earlier in this chapter, you saw how to retrieve the current browser locale on the client side and how to set the global locale of your context or session to this value. Once the appropriate locale has been set, you can easily format dates using FormatDateTime in VBScript, a locale-aware function. Suppose you retrieve Spanish (Mexico) as the primary browser locale. ("es-MX" is the default value for this locale.) The following code saves the current context locale (matching the server's user locale), sets the locale to Spanish (Mexico), formats the date in Spanish (Mexico) format, and restores the original locale:

currentLocale = GetLocale
Original = SetLocale("es-MX")
DateData = FormatDateTime(Date(),vbLongDate)
Original = SetLocale(currentLocale)

And the output would be:

martes, 18 de diciembre de 2001

Obviously, scripting technology does not offer the same flexibility to manipulate dates and calendars as NLS APIs do in the case of Win32 programming. However, the FormatDateTime function lets you display dates in the user's preferred culture, in both short-date and long-date formats.

Date and Calendar Formatting in the .NET Framework

The easiest and most efficient way of formatting dates in the .NET world is to take advantage of the DateTime structure, which provides methods allowing you to perform culture-sensitive operations on DateTime. Use the DateTimeFormatInfo class to format and display a DateTime based on a culture-specific standard. DateTimeFormatInfo defines how DateTime values are formatted and displayed, depending on the culture. For example, using the ShortDatePattern, the date April 24, 2001, is formatted as 4/24/2001 for the "en-US" culture and 24/04/2001 for the "en-GB"-English (United Kingdom)-culture.

An instance of DateTimeFormatInfo can be created for a specific culture, but not for a neutral culture. A neutral culture does not provide enough information to display the correct date format. You just saw that English (United States) and English (United Kingdom) share the same neutral culture-English-and yet have completely different formatting for the date.

The following code example displays the current date using the DateTime-FormatInfo.ShortDatePattern when the CurrentThread.CurrentCulture is set to "en-US," and then to "de-DE" (German in Germany).

using System;
using System.Globalization;
using System.Threading;

public class FormatDate
{
   public static void Main()
   {      DateTime dt = DateTime.Now;
      // Set the CurrentCulture property to U.S. English.
      Thread.CurrentThread.CurrentCulture = new CultureInfo("en-US");

      // Display dt, formatted using the ShortDatePattern.
      //    and the CurrentThread.CurrentCulture.
      Console.WriteLine(dt.ToString("d"));

      // Create a CultureInfo object for German in Germany.
      CultureInfo ci = new CultureInfo("de-DE");

      // Display dt, formatted using the ShortDatePattern
      //    and the CultureInfo object.
      Console.WriteLine(dt.ToString("d", ci));
   }
}

If you execute this code on April 24, 2001, the output appears as follows:

4/24/2001         // short-date format for English (United States)
24.04.2001        // short-date format for German (Germany)

Table 4-3 lists the standard format characters for each standard pattern of formatting, and the associated DateTimeFormatInfo property that can be set to modify the standard pattern. The format characters are case-sensitive; for example, "g" and "G" represent slightly different patterns.

Table 4-3 List of DateTimeFormatInfo properties, patterns, and date formats for "en-US" format patterns. The format pattern varies according to culture.
Format Character | Format Pattern | Associated Property/Description
d | MM/dd/yyyy | ShortDatePattern
D | dddd, dd MMMM yyyy | LongDatePattern
f | dddd, dd MMMM yyyy HH:mm | Full date and time (long date and short time)
F | dddd, dd MMMM yyyy HH:mm:ss | FullDateTimePattern (long date and long time)
g | MM/dd/yyyy HH:mm | General (short date and short time)
G | MM/dd/yyyy HH:mm:ss | General (short date and long time)
m, M | MMMM dd | MonthDayPattern
r, R | ddd, dd MMM yyyy HH':'mm':'ss 'GMT' | RFC1123Pattern
s | yyyy'-'MM'-'dd'T'HH':'mm':'ss | SortableDateTimePattern (based on ISO 8601) using local time
t | HH:mm | ShortTimePattern
T | HH:mm:ss | LongTimePattern
u | yyyy'-'MM'-'dd HH':'mm':'ss'Z' | UniversalSortableDateTimePattern (based on ISO 8601) using uni- versal time
U | dddd, dd MMMM yyyy HH:mm:ss | Sortable date and time (long date and long time) using universal time
y, Y | yyyy MMMM | YearMonthPattern

In Thailand, the Gregorian and the Buddhist calendar coexist. Internally, however-similar to the Win32 paradigm-the .NET Framework handles dates in the Gregorian calendar by using data structures. Therefore, when you use the methods provided by the DateTime structure, you must be aware that the members such as the DateTime.Day property, the DateTime.Month property, the DateTime.Year property, and the DateTime.AddDays method are based on the Gregorian calendar. Even if you change the current calendar in your application's code or change date and time settings through the Regional And Language Options property sheet, the Gregorian calendar is still used to perform the calculations for these methods. This functionality prevents the arithmetic performed by these methods from being corrupted by a user's settings. If you want to perform culture-sensitive date and time operations based on the current calendar, you must use the DateTimeFormatInfo.Calendar property to call methods provided by the Calendar class such as Calendar.GetDayOfMonth, Calendar.GetMonth, Calendar-.GetYear, and Calendar.AddDays.

To handle native calendar types, the .NET Framework provides the Calendar class as well as the following Calendar implementations: GregorianCalendar, HebrewCalendar, HijriCalendar, JapaneseCalendar, JulianCalendar, KoreanCalendar, TaiwanCalendar, and ThaiBuddhistCalendar. Other things you can use include the CultureInfo class, which has a CultureInfo.Calendar property that specifies a culture's default calendar. The CultureInfo.OptionalCalendars property specifies the optional calendars supported by a culture.

The following code example in C# creates CultureInfo objects for the culture "th-TH" (Thai in Thailand) and displays the default and optional calendars for each culture. The GregorianCalendar is further divided into subtypes by the GregorianCalendar.CalendarType property. In this example, each time the calendar is determined to be Gregorian, the CalendarType value is retrieved and displayed.

using System;
using System.Globalization;
public class TestClass
{
   public static void Main()
   {
      // Create a CultureInfo object for Thai in Thailand.
      CultureInfo th = new CultureInfo("th-TH");
      DisplayCalendars(th);
   }

   protected static void DisplayCalendars(CultureInfo cultureinfo)
   {
      CultureInfo ci = new CultureInfo(cultureinfo.ToString());

      // Display the default calendar for the culture.
      if (ci.Calendar is GregorianCalendar)
         Console.WriteLine ("\n\n");
         Console.WriteLine (
            "The default calendar for the {0} culture is: \n{1}\n\n",
            ci.DisplayName.ToString(), ci.Calendar.ToString() +
            " subtype " +
            ((GregorianCalendar)ci.Calendar).CalendarType.ToString());
      else
         Console.WriteLine ("\n\n");
         Console.WriteLine (
           "The default calendar for the {0} culture is: \n{1}\n\n",
           ci.DisplayName.ToString(), ci.Calendar.ToString());
      // Display the optional calendars for the culture.
        Console.WriteLine (
         "The optional calendars for the {0} culture are: ",
         ci.DisplayName.ToString());
      for (int i = ci.OptionalCalendars.GetLowerBound(0); i <=
         ci.OptionalCalendars.GetUpperBound(0); i++ )
         {

            if (ci.OptionalCalendars[i] is GregorianCalendar)
            {
               // Display the calendar subtype.
               String CalStr = (ci.OptionalCalendars[i].ToString() +
                  " subtype " +
((GregorianCalendar)ci.OptionalCalendars[i]).CalendarType.ToString());
               Console.WriteLine(CalStr);
            }
            else
               Console.WriteLine (ci.OptionalCalendars[i].ToString());
         }
   }
}

This code produces the following output:

The default calendar for the Thai (Thailand) culture is:
      System.Globalization.ThaiBuddhistCalendar

The optional calendars for the Thai (Thailand) culture are:
   System.Globalization.ThaiBuddhistCalendar
               System.Globalization.GregorianCalendar
 subtype Localized

As you see, the .NET Framework offers a UI that's extensive, yet flexible and easy to use for date and calendar formatting. The sections that follow outline the points to consider when representing time in Win32 applications, Web pages, and the .NET Framework. Included is information on using the appropriate separators between hours, minutes, and seconds; constructing format-picture strings; and displaying time zones.


Time Formatting

Like date and calendar formats, time formats are not constant throughout the world. Although each representation of time basically displays the hour, minutes, and seconds, their presentation order and separators vary greatly. In fact, there might be many differences between regions within the same country. The time formatting might differ from one culture to another in one of the following three ways:

The use of either a 12-hour or 24-hour clock. Most European and Asian locales use the 24-hour clock instead of the 12-hour A.M./P.M. model used in the United States. Also, A.M./P.M. can be rendered in the language of the country, and in some languages comes before the time and not after it.

The character used to separate hours, minutes, and seconds. Although the colon (:) is the character most used in separating the hours, minutes, and seconds, some Asian languages use ideographic characters. In addition, some locales require "h," "m," and "s" as part of the display.

The storage and display of time zones. One way that's frequently used to represent the time zone is to display GMT (Greenwich Mean Time) or its modern replacement UTC (Coordinated Universal Time) as the base. This is then followed by the time zone, which is indicated as a positive or negative offset in hours and minutes. (Some time zones use 30-minute or 45-minute offsets.) For example, take Bangalore, India, for which the time zone would be displayed as UTC +5:30, or Chatham Island, New Zealand, for which it would be UTC +12:45. Another way to display time zones is by using names for the local time zones. If you do so, you must take the following into account:

  • Not all countries use local names.
  • Time-zone abbreviations are not unique.
  • Not all countries use daylight saving time, and daylight saving time does not start and end on the same day in every country.
  • One time zone might have many different names, depending on the country and the language.

The user can define the preferred time format, time separator, and A.M./P.M. symbols by making selections from the Time tab of the Customize RegionalOptions property sheet, within the Regional And Language Options property sheet. (See Figure 4-14 .)

The user can select the current time zone from the Time Zone tab of the Date And Time Properties property sheet. (See Figure 4-15.)

Important Changing the user locale or the locale variable does not adjust the time zone by default.
Figure 4-14 Time formatting for Greek user locale.
Figure 4-14 - Time formatting for Greek user locale.
Figure 4-15 Time Zone page of Date And Time Properties property sheet.
Figure 4-15 - Time Zone page of Date And Time Properties property sheet.

Time Formatting and Time Zones in Win32

To format time in the default settings of a given locale or as specified by the user in the Regional And Language Options property sheet, you can use GetTimeFormat. This function formats time-either a specified time or the local system time-as a time string for a specified locale.

The following code sample displays the current system time for the current user locale, using the default time format for that locale:

// Formats time as a time string for a specified locale.
GetTimeFormat(LOCALE_USER_DEFAULT,  // predefined current user locale
   0,          // option flag for things like no usage of seconds or
               //    force 24h clock
   NULL,       // time - NULL to go with the current system locale time
   NULL,       // time format string - NULL to go with default locale
               //    format
   g_szBuff1,  // formatted string buffer
   MAX_STR);   // size of string buffer

Execution of this code would give the following result on English (United States) and Punjabi user locales, respectively. (See Figure 4-16.)

Figure 4-16 Time formatted for English (United States) and Punjabi user locales.
Figure 4-16 - Time formatted for English (United States) and Punjabi user locales.

As you saw in the previous example, the time formatting can be completely different from one locale to another. In this case, the use of a leading 0 in the hour representation, as well as the actual translation and positioning of P.M., change based on country and cultural standards. But even within the same locale or culture there is a variety of possible ways to format time: short or long formatting. The EnumTimeFormats function enumerates the time formats that are available for a specified locale. It does so by passing (to a callback function that an application defines) a pointer to a buffer that contains a time format. The EnumTimeFormats continues to do so until no more time formats are found, or until the callback function returns FALSE.

Here is how the code works:

EnumTimeFormats(EnumTimeFormatsProc, // enumeration callback function
   LOCALE_USER_DEFAULT,              // locale for which the
                                    // enumeration is done
   NULL);                            // unused

// The callback function...
BOOL CALLBACK EnumTimeFormatsProc(LPTSTR lpTimeFormatString)
   {
   // end of enumeration, then terminate
   if (!lpTimeFormatString)
         return FALSE;
         MessageBox(NULL, g_lpTimeFormatString,
         TEXT("Available Time format"), MB_OK);

      return TRUE;
}

Execution of this code would give the following result on English (United States) and French (France) user locales, respectively. (See Figure 4-17.)

Figure 4-17 Time formats for English (United States) and French (France) user locales.
Figure 4-17 - Time formats for English (United States) and French (France) user locales.

The obtained result (format picture) can then be used as an lpFormat argument in a call to GetTimeFormat. This allows you to use an alternative formatting for the time. The following elements can be used to construct a format-picture string. (See Table 4-4.) If you use spaces to separate the elements in the format string, these spaces will appear in the same location in the output string. The letters must adhere to the casing conventions shown in Table 4-4. (For example, the correct formatting for "Seconds with leading zero for single-digit seconds" would be "ss" not "SS".) Characters in the format string that are enclosed in single quotation marks will appear in the same location and will be unchanged in the output string.

Table 4-4 Elements for constructing a format-picture string.
Picture | Meaning
h | Hours with no leading zero for single-digit hours; 12-hour clock.
hh | Hours with leading zero for single-digit hours; 12-hour clock.
H | Hours with no leading zero for single-digit hours; 24-hour clock.
HH | Hours with leading zero for single-digit hours; 24-hour clock.
m | Minutes with no leading zero for single-digit minutes.
mm | Minutes with leading zero for single-digit minutes.
s | Seconds with no leading zero for single-digit seconds.
ss | Seconds with leading zero for single-digit seconds.
t | One character time-marker string, such as A or P.
tt | Multicharacter time-marker string, such as A.M. or P.M.

For example, to get the time string:

"11:29:40 PM"

use the following picture string:

"hh':'mm':'ss tt"

To retrieve information regarding the time parameters for a particular time zone that the user has selected, you can use the GetTimeZoneInformation function. These parameters control the translations between UTC and local time.

DWORD GetTimeZoneInformation(
      LPTIME_ZONE_INFORMATION lpTimeZoneInformation);
// Where LPTIME_ZONE_INFORMATION is defined as:

typedef struct _TIME_ZONE_INFORMATION
{
   LONG Bias;
   WCHAR StandardName[ 32 ];
   SYSTEMTIME StandardDate;
   LONG StandardBias;
   WCHAR DaylightName[ 32 ];
   SYSTEMTIME DaylightDate;
   LONG DaylightBias
}  TIME_ZONE_INFORMATION, *PTIME_ZONE_INFORMATION;

The Bias parameter of the TIME_ZONE_INFORMATION structure is the difference, in minutes, between UTC time and local time.

All translations between UTC time and local time are based on the following formula:

UTC = local time + bias

Unfortunately, Win32 APIs do not offer a solution to enumerate information relative to any other time zone than the one currently selected by the user. GetLocalTime and GetSystemTime will allow you to retrieve the current local and the UTC times, respectively. To convert the local time to UTC, you can use the LocalFileTimeToFileTime API. These are but two of the many Win32 APIs dealing with time. To learn about other functions, search for "time functions" on http://msdn.microsoft.com.

Time Formatting in Web Pages

In "Retrieving the Browser Language Setting" earlier in this chapter, you saw how to retrieve the current browser locale on the client side and how to set the global locale of your context or session to this value. Once the appropriate locale has been set, you can easily format the time using FormatDateTime, a locale-aware function. Suppose you have retrieved the Chinese (Taiwan) locale as the primary browser locale ("zh-TW" is the default value for this locale). The following code saves the current context locale (matching the server's user locale), sets the locale to Chinese (Taiwan), formats the date in Chinese (Taiwan) format, and restores the original locale.

currentLocale = GetLocale
Original = SetLocale("zh-TW")
DateData = FormatDateTime(Now(),vbLongTime)
Original = SetLocale(currentLocale)

And the output would be:

   04:16:08

Obviously the scripting technology does not offer the same flexibility to manipulate time formats as NLS APIs do in the case of Win32 programming. However, the FormatDateTime function gives you the ability to display time in the user's cultural preferences in both short- and long-time formats.

Time Formatting and Time Zones in the .NET Framework

The easiest and most efficient way of doing time formatting in the .NET world is to take advantage of the DateTime structure that provides methods such as DateTime.ToString and DateTime.Parse. These methods allow you to perform culture-sensitive operations on a DateTime object. Use the DateTimeFormatInfo class to format and display a DateTime based on culture. DateTimeFormatInfo defines how DateTime values are formatted and displayed, depending on the culture. For example, using the LongTimePattern, the time 4 hours P.M., 36 minutes and 15 seconds is formatted as 4:36:15 PM for the "en-US" culture and 16:36:15 for the "fr-FR" culture number-formatting standards. (For more information and code samples, see "Date and Calendar Formatting in the .NET Framework" earlier in this chapter.)

Methods and properties in the DateTime structure always use the local time zone for calculations and comparisons. You should consider this when using the DateTime.Parse method and the DateTime.ParseExact method. These methods provide overloads that allow you to convert the string representation of a date and time to a DateTime type. You can also choose to format a DateTime for a specific culture. If you do not specify a time zone in the string that you pass to these methods, they return the parsed date and time without performing a time-zone adjustment. The date and time are based on the system's time-zone setting. If you specify a time-zone offset, these methods parse the date/time string, convert it to UTC, and then convert it to the time on the local system.

Use the DateTime.ToUniversalTime method to convert a local DateTime to its UTC equivalent. To parse a date/time string and convert it to a UTC DateTime, use the DateTimeStyles enumeration AdjustToUniversal value with either the DateTime.Parse or DateTime.ParseExact method. These DateTime manipulations are illustrated in the following code example. This example creates a DateTime for the local time and then converts it to the UTC equivalent DateTime. Both types are converted to strings and written to the console. Notice that the strings differ by the UTC offset between the local time zone and UTC. (For more information on the UTC offset for various time zones, see the TimeZone.GetUtcOffset method in the Microsoft Developer Network [MSDN] documentation at http://msdn.microsoft.com.) These strings are converted back to DateTime types using the DateTime.ParseExact method. To capture the time-zone information stored in utcdt, the AdjustToUniversal value must be specified as a parameter to the DateTime.ParseExact method.

using System;
using System.Globalization;
using System.Threading;

public class TimeZoneSample
{

  public static void Main()
   {
      CultureInfo en = new CultureInfo("en-US");
      Thread.CurrentThread.CurrentCulture = en;

      // Create a DateTime object for the local time.
      DateTime dt = new DateTime(2001, 7, 13, 4, 0, 0);

      // Convert the local DateTime to the UTC time.
      DateTime utcdt = dt.ToUniversalTime();

      // Define a custom string format to display the DateTime
      //    value. zzzz specifies the full time zone offset.
      String format = "MM/dd/yyyy hh:mm:sszzz";

      // Convert the local DateTime object to a string
      //    using the custom format string and display.
      String str = dt.ToString(format);
      Console.WriteLine(str);

      // Convert the UTC DateTime to a string
      //    using the custom format string and display.
      String utcstr = utcdt.ToString(format);
      Console.WriteLine(utcstr);

      // Convert the string back to a local DateTime object and
      //    display.
      DateTime parsedBack =
         DateTime.ParseExact(str,format,en.DateTimeFormat);
      Console.WriteLine(parsedBack);

      // Convert the string back to a UTC DateTime object and
      //    display. If you do not use the DateTime.ParseExact
      //    method that takes a DateTimeStyles.AdjustToUniversal
      //    value, the parsed DateTime object will not include the
      //    time zone information.

      DateTime parsedBackUTC = DateTime.ParseExact(str, format,
         en.DateTimeFormat, DateTimeStyles.AdjustToUniversal);
      Console.WriteLine(parsedBackUTC);
   }
}

This code produces the following output:

07/13/2001 04:00:00-07:00
07/13/2001 11:00:00-07:00
7/13/2001 4:00:00 AM
7/13/2001 11:00:00 AM

Another vital point to consider when creating an application that is locale-aware is how currency is displayed. Since money is a valued commodity, it is essential that such things as the correct currency symbol as well as the appropriate decimal and thousands separator be used, for example, or the consequences could potentially be disastrous!


Currency Formatting

A locale-aware and culture-sensitive representation of currency needs to take into consideration the following elements:

Currency symbol. This can be a predefined symbol like the European euro " [image] " or a combination of letters like the use of "lei" for the Romanian's Lei.

Currency symbol placement. It can be placed either before or after the digits.

Negative-amount display. There are several ways to display negative amounts. (See Table 4-5.)

Most currencies use the same decimal and thousands separators that the numbers in the locale use, but this is not always true. In Estonia, they use the period as a decimal separator for Estonian Kroons (127.54 kr), but then use commas as the decimal separator for everything else (127,54).

Table 4-5 Currency conventions in various locales for negative amounts.

Currency Convention | Locale | Result
The negative sign before both the currency symbol and number | U.K. France | -£127.54 -127,54
The negative sign before the number but behind the currency symbol | Denmark | kr-127,54
The negative sign after the number | Netherlands | 127,54 [image] -
The use of parentheses | U.S. | ($127.54)

The user can define the preferred currency symbol as well as the positive and negative currency formats by making selections from the Currency tab of the Customize Regional Options property sheet, within the Regional And Language Options property sheet. (See Figure 4-18.)

Important
All techniques described in this section about handling currency formats in Win32, Web pages, and the .NET Framework only deal with the actual representation of data. The system does not offer a conversion mechanism between different monetary units. It's up to developers to implement that extra step. For example, if you decide to format $1,000 into Dutch (Belgium) standards, the system will return to you 1.000 [image], even though $1 is not equivalent to 1 [image].
Figure 4-18 Currency formatting for China (PRC).
Figure 4-18 - Currency formatting for China (PRC).

Currency Formatting in Win32

Win32 NLS APIs can help you display currency data in a way that's locale-aware. The GetCurrencyFormat function formats a number string as a currency string for a specified locale. The code sample on the following page formats a number string into the user locale's default format:

GetCurrencyFormat(LOCALE_USER_DEFAULT, // a predefined value for
                                       //    user locale
   NULL,               // operation option
   TEXT("123.40"),     // input number (see MSDN for accepted chars)
   NULL,               // formatting specifications
   g_szTemp,           // output buffer
   MAX_STR);           // size of output buffer

Execution of the previous code would give the following result on English (United States) and Danish user locales, respectively. (See Figure 4-19.)

Figure 4-19 Currency format for English (United States) and Danish.
Figure 4-19 - Currency format for English (United States) and Danish.

With this approach, the number string is formatted to a locale-specific format and, more importantly, currency symbol! But suppose you want to represent $1,230.40 (U.S. dollars) to a Danish user. By using the first approach mentioned and formatting your string with current user locale settings (Danish), your string would in fact come out as kr 1.230,40. Of course, the main issue would be that you want to format your string in the Danish format, but you don't want to change your currency symbol (and in this case lose some of the value of your money in the transaction).

Another scenario would be the use of the euro as the official currency of Belgium, Germany, Spain, France, Ireland, Italy, Luxembourg, the Netherlands, Austria, Portugal, Finland, and Greece. (This list is as of January 1, 2002.) The .NET Framework and Windows XP set the default currency symbol to the euro for these twelve countries that use the euro as their official currency. However, older versions of Windows, without any intervention, will still set the default currency symbol to the local currency for these countries or regions. To resolve the issue in the two scenarios just mentioned, you can take advantage of the lpFormat argument of GetCurrencyFormat. This argument is a pointer to a currency-formatting structure, where you can define your own formatting model and specify the currency symbol to be used following the formatting standard of a given locale. The following code sample uses the formatting for the currently selected user locale, but with the euro as the desired currency symbol. Here is how it works.

CURRENCYFMT CurFormat;

// First fill in the CURRENCYFMT structure with user locale-
specific
//    information.
GetLocaleInfo(LOCALE_USER_DEFAULT, LOCALE_RETURN_NUMBER|LOCALE_IDIGITS,
   &CurFormat.NumDigits, STR_LEN);
GetLocaleInfo(LOCALE_USER_DEFAULT, LOCALE_RETURN_NUMBER|LOCALE_ILZERO,
   CurFormat.LeadingZero, STR_LEN);
GetLocaleInfo(LOCALE_USER_DEFAULT, LOCALE_SGROUPING,
   &CurFormat.Grouping, STR_LEN);
GetLocaleInfo(LOCALE_USER_DEFAULT, LOCALE_SDECIMAL,
   &CurFormat.lpDecimalSep, STR_LEN);
GetLocaleInfo(LOCALE_USER_DEFAULT, LOCALE_STHOUSAND,
   &CurFormat.lpThousandSep, STR_LEN);
GetLocaleInfo(LOCALE_USER_DEFAULT, LOCALE_RETURN_NUMBER|LOCALE_INEGCURR,
   CurFormat.NegativeOrder, STR_LEN);
GetLocaleInfo(LOCALE_USER_DEFAULT,
   LOCALE_RETURN_NUMBER|LOCALE_ICURRENCY, CurFormat.PositiveOrder,
   STR_LEN);

// Set euro as the default currency symbol.
CurFormat.lpCurrencySymbol = TEXT("  ");

GetCurrencyFormat(LOCALE_USER_DEFAULT, // a predefined value for
                                       //    user locale
   0,                    // operation option
   TEXT("123.40"),       // input number (see MSDN for legal chars)
   &CurFormat,           // formatting specifications
   g_szTemp,             // output buffer
   MAX_STR);             // size of output buffer

By following the code, the result of a currency symbol formatting for the Danish locale (with kr as default currency symbol) would be: 123,40

Currency Formatting in Web Pages

In "Retrieving the Browser Language Setting" earlier in this chapter, you saw how to retrieve the current browser locale on the client side and how to set the global locale of your context or session to this value. Once the appropriate locale has been set, you can easily format currency by using FormatCurrency, a locale-aware function. Suppose you have retrieved Bulgarian as the primary browser locale ("bg" being the default value for this locale). The following code saves the current context locale (matching the server's user locale), sets the locale to Bulgarian, formats the date in Bulgarian format, and restores the original locale.

currentLocale = GetLocale
Original = SetLocale("bg")
DateData = FormatCurrency(3943.23)
Original = SetLocale(currentLocale)

And the output would be: 3 943,23

Obviously the scripting technology does not offer the same flexibility to manipulate currency symbols as NLS APIs do in the case of Win32 programming. However, theFormatCurrency function gives you the ability to display currency according to the user's cultural preferences, in both short and long currency formats.

Currency Formatting in the .NET Framework

Glossary
  • Windows Forms: The new platform for Microsoft Windows application development, based on the .NET Framework. Provides a clear, object-oriented, extensible set of classes that enables you to develop rich Windows applications. Additionally, Windows Forms can act as the local UI in a multitier, distributed solution.
  • ISO 4217 currency symbol: Three-letter ISO codes for representing currencies and funds (such as CAD for the Canadian dollar).

The .NET format strings are useful if you want to convert one of the standard .NET Framework data types to a string that represents that type in some other format. For example, if you have an integer value of 100 that you want to represent to the user as a currency value, you could easily use the ToString method and the currency-format string ("C") to produce a string of "$100.00". Computers that do not have English (United States) specified as the current culture will display whatever currency notation is used by the current culture. The original value contained in the data type is not converted, but a new string is returned that represents the resulting value. This new string cannot be used for calculation until it is converted back to a .NET base data type. The original value, however, can be calculated at any time.

In the following code example, the ToString method displays the value of 100 as a currency-formatted string in the console's output window.

int MyInt = 100;
String MyString = MyInt.ToString("C");
Console.WriteLine(MyString);

This code displays $100.00 to the console on computers that have English (United States) as the current culture.

The CurrencySymbol property of the RegionInfo class from the System-.Globalization namespace can be used to retrieve the currency symbol associated with the country or region. Also, the ISOCurrencySymbol property of RegionInfo retrieves the three-character ISO 4217 currency symbol associated with the country or region.

Regarding the euro challenge mentioned earlier, again, the euro is now the default currency symbol of twelve European countries, but only Windows XP has this currency symbol set as the default for the affected countries. On downlevel platforms (including Windows 2000), this value has to be set manually by the user. If you want to ensure that your application uses the .NET Framework's default settings and use the euro as the default currency symbol (regardless of the user settings), you must create a CultureInfo object, passing a useUserOverride parameter set to false.

The following example uses code that is similar to the previous example. It sets the current culture to "fr-FR" and displays an integer to the console formatted as currency. The user's local currency settings are used to format the currency. Next, the culture is set to "fr-FR" using the CultureInfo constructor that accepts the useUserOverride parameter set to false. The number is then formatted using the .NET Framework default settings, and the euro currency symbol is displayed.

using System;
using System.Globalization;
using System.Threading;

public class EuroSymbolSample
{
   public static void Main()
   {
      int i = 100;

      // Set the CurrentCulture to French in France.
      Thread.CurrentThread.CurrentCulture =
            new CultureInfo("fr-FR");

      // Display i formatted as default currency for the
      //    CurrentCulture.
      // On a version of Windows prior to Windows XP, where the user
      //    has not changed the default currency to euro through
      //    Control Panel, this will default to "F".
      Console.WriteLine(i.ToString("c"));

      // Set the CurrentCulture to French in France, using the
      //    CultureInfo constructor that takes a useUserOverride
      //    parameter.
      // Set the useUserOverride value to false.
      Thread.CurrentThread.CurrentCulture =
         new CultureInfo("fr-FR", false);
// Display i formatted as default currency for the
      //    CurrentCulture.
      // On a version of Windows prior to Windows XP, this will
      //    override an incorrect default setting of "F" and display
      //    the euro symbol (i).
      Console.WriteLine(i.ToString("c"));
   }
}

If you execute this code in a Windows Forms application, the output appears as follows:

    100,00 F

    100,00

Although most European Union countries now use the euro, there might be situations-such as for legacy reasons-where it is necessary to display both the euro and the older currencies in an application.The following code example creates a CultureInfo object for the culture "fr-FR" where the default currency is the euro. To display the currency symbol for the local currency, you must use the NumberFormatInfo.Clone method to clone a new NumberFormatInfo object for the CultureInfo object and replace the default currency symbol with a local currency symbol.

using System;
using System.Globalization;
using System.Threading;
public class EuroLocalSample
{public static void Main()
   {
      // Create a CultureInfo object for French in France.
      CultureInfo FrCulture = new CultureInfo("fr-FR");
      // Set the CurrentCulture to fr-FR.
      Thread.CurrentThread.CurrentCulture = FrCulture;
      // Clone the NumberFormatInfo object and create
      //    a new object for the local currency of France.
      NumberFormatInfo LocalFormat =
         (NumberFormatInfo)NumberFormatInfo.CurrentInfo.Clone();
      // Replace the default currency symbol with the local
      //    currency symbol.
      LocalFormat.CurrencySymbol = "F";

      int i = 100;

      // Display i formatted as the local currency.
      Console.WriteLine(i.ToString("c", LocalFormat));
      // Display i formatted as the default currency.
      Console.WriteLine(i.ToString("c", NumberFormatInfo.CurrentInfo));
   }
}

The following section will help you handle linguistic uppercasing and lowercasing rules that correspond to the currently selected user-locale value. As you'll see, whether you represent something as uppercase or lowercase might initially seem insignificant, but there can be considerable consequences to ignoring casing rules.


Casing

Glossary
  • Case-folding: Taking a string of text and converting everything into either lowercase or uppercase.
  • Lowercase: Denotes letters that are not capitalized. For instance, the word "nationality" is all lowercase. The notion of lowercase does not apply to East Asian and Middle Eastern scripts.
  • Uppercase: Denotes letters that are capitalized. For instance, acronyms (such as "HTML") typically consist of all uppercase letters. The notion of uppercase does not apply to East Asian and Middle Eastern scripts.

Another thing you'll need to consider when creating a locale-aware application involves linguistic nuances. These nuances might seem trivial, but could have a large impact on application design and functionality. For example, Windows allows you to convert characters into either uppercase or lowercase equivalents. Some applications use this feature to automatically convert the first letter of every sentence into uppercase or to assume that certain types of words should always be capitalized. In Russian, however, names of the days of the week are never capitalized-capitalizing the word for "Wednesday" changes the meaning to "environment," and capitalizing the word for "Sunday" changes the meaning to "resurrection."

In the past as localized products were developed, language-sensitive issues-such as casing-were sometimes handled with what were thought of as well-designed, intelligent algorithms. For example, an uppercasing macro that relies onthe code-point numbers of ASCII characters and the linear relationship between uppercase characters (A = 41) and lowercase characters (a = 61) can be written as:

#define ToUpper(ch)  ((ch)<='Z' ? (ch) : (ch)+'A' - 'a')

You can see the problems this English-centric approach presented when representing uppercasing on non-Latin scripts or languages with accented characters where, for example, character mapping doesn't follow the assumed relationship between lowercase and uppercase characters? There are several other reasons why algorithmic solutions for case-folding do not cover all occurrences.

First, some languages do not have a one-to-one mapping between their uppercase and lowercase characters. For instance, the uppercase equivalent of the German ß is "SS." Second, some characters have different mappings depending upon the language in which they are used. For example, the lowercase "i" in English maps to a dotless uppercase letter: "I." However, in Turkish the lowercase "i" maps to a dotted uppercase letter: " [image]." Finally, most non-Latin scripts do not even use the concept of lowercase and uppercase, as in the case of Chinese, Japanese, and Korean; Arabic, Farsi, and Hebrew; as well as Thai. For example, since Farsi has no notion of uppercasing, string output is composed of random and unsup-ported glyphs. (See Figure 4-20.)

Figure 4-20 The English-centric uppercasing macro used on an English string and on a Farsi string, where the notion of casing does not exist.
Figure 4-20 - The English-centric uppercasing macro used on an English string and on a Farsi string, where the notion of casing does not exist.

Casing in Win32

CharUpper and CharUpperBuff functions convert lowercase characters of a string or a buffer, respectively, to uppercase characters. This uppercasing is done with regard to the currently selected user-locale value and the linguistic uppercasing rules associated with this locale. CharLower and CharLowerBuff functions convert uppercase characters of a string or a buffer, respectively, to lowercase characters. This lowercasing is done with regard to the currently selected user-locale value and the linguistic lowercasing rules associated with this locale.

If you want to perform the casing operation based on locale standards other than the currently selected user-locale rules (something that the functions just mentioned do not allow), you can use LCMapString as shown here:

LCMapString(
   LCID Locale,        // locale identifier whose rule will be used
                       //    to perform the casing
   DWORD dwMapFlags,   // mapping transformation (LCMAP_LOWERCASE or
                       //    LCMAP_UPPERCASE
   LPCTSTR lpSrcStr,   // source string
   int cchSrc,         // number of characters in source string
   LPTSTR lpDestStr,   // destination buffer
   int cchDest         // size of destination buffer
);

Casing in Web Pages

Scripts running in a browser might need to manipulate the character casing. VBScript and Microsoft JScript both provide means for case conversions thatoperate on multilingual input. In VBScript, use UCase to convert a string to uppercase and LCase to convert a string to lowercase; in JScript, use String.toUpperCase and String.toLowerCase, respectively.

The following example uses the UCase function to return an uppercase version of a string:

Dim MyWord
MyWord = UCase("Hello World")   ' Returns "HELLO WORLD".

Obviously the scripting technology does not offer the same flexibility to manipulate the casing as NLS APIs do in the case of Win32 programming. However, the functions available give you the ability to manipulate the casing for whatever string, though the conversion is not locale-sensitive.

Casing in the .NET Framework

The String class provides a set of methods you can use to perform culture-sensitive string manipulation once you set your CurrentCulture to a given desired culture. The String.ToUpper and String.ToLower methods can be used to convert a character string to uppercase or lowercase for a given culture.

Another major area that pertains to locale awareness is sorting in a way that matches the particular locale. The sections that follow examine the best way to accommodate the multiple sort orders that exist in various countries and regions. They will also illustrate the most efficient ways to perform string comparison for Win32 applications and in the .NET Framework.


Sorting and String Comparison

Glossary
  • Radicals: A group of strokes in a Chinese character that are treated as a unit for the purposes of sorting, indexing, and classification. A character can contain more than one element that is recognized as a radical, but each character contains only one element, called the "main radical," that is used as the indexing radical. The main radical often gives a hint as to the general meaning of the character, and other radicals in the character might indicate how the character is pronounced.
  • Sort keys: Numeric representations of a sort element based on locale-specific sorting rules. A sort key consists of several weighted components that represent a character's script, diacritics, case, and so on.

Even though you might think you understand all the issues involved with sorted lists, users of world-ready applications might have very different expectations of what constitutes a "sorted" list. Not only does alphabetical order vary among languages, but conventions for sequencing items in dictionaries and phone books can also be quite different.

In Swedish, for example, some vowels with an accent sort after "Z," whereas in other European countries the same accented vowel comes right after the nondiacritic vowel. Languages that include characters outside the Latin script have special sorting rules. The Asian languages have several different sort orders depending on phonetics, radical order, number of pen strokes, and so on.

String Comparison and Sorting in Win32

String sorting and comparison are language-specific. Even within languages based on the Latin script, there are different composition and sorting rules. Thus do not rely on code points to do proper sorting and string comparison.

String Comparison

The lstrcmp and lstrcmpi Win32 functions compare two character strings in a case-sensitive or case-insensitive manner, respectively, following the sorting rules and standards that are defined for the currently selected user locale. The lstrcmp and lstrcmpi functions compare two strings by checking the first characters against each other, the second characters against each other, and so on until they find an inequality or reach the ends of the strings. These functions return the difference of the values of the first unequal characters they encounter. For example, lstrcmp determines that "abcz" is greater than "abcdefg" and returns the difference of the code-point numbers that correspond to "z" and "d." The selected user locale determines which string is greater (or whether the strings are the same).

The big disadvantage of these APIs is that you cannot specify which locale-specific rules should be used to perform the comparison, and you are always limited to the currently selected user-locale standards. That can be a problem in the following cases:

  • You want to display data in a client context and, therefore, you need to perform your comparison within the client-side, user-locale specifications. You want to do a string comparison that's locale-aware, but you'd like to use the rules of a locale different from those of your current user locale. The CompareString API allows you to compare two character strings and to specify a locale to be used to do this comparison.
  • To prevent locale-specific composition or sorting rules from affecting the result of your comparison, you do a comparison that is not dependent upon locale. Imagine that you retrieve some kind of data fromthe registry (such as "HorzScroll") and want to compare that data against a predefined list of items (such as "HORZSCROLL") by using lctrcmpi. This performance being internal to your application, you always want to perform your comparison within the same condition. But unfortunately, "sc" is a special composition within the Hungarian language (as opposed to "Sc"). Thus your comparison would always fail if the user locale is set to Hungary! In this case, what you should be doing is a locale-insensitive (or locale-independent) comparison.

Windows XP introduced a new locale ID named the "invariant locale"-LOCALE_INVARIANT is the predefined flag for it-that allows you to do this type of comparison. A locale-independent string comparison must use this new locale ID.

CompareString(LOCALE_INVARIANT, NORM_IGNORECASE, TEXT("HorzScroll"),
   -1, TEXT("HorzScroll"), -1);

By using the invariant locale, you are always sure to retrieve the same comparison result no matter what the user locale and its associated sorting rules are. However, the invariant locale is only available on Windows XP. Therefore, for downlevel platforms you'll need to define your own locale-such as English-and then in this instance use basic comparison rules of the English language that follow the ASCII mapping of characters. Here is how the code should look:

LCID Locale;
Locale = MAKELCID(MAKELANDIG (LANG_ENGLISH, SUBLANG_ENGLISH_US),
            SORT_DEFAULT);
ComapareString(Locale, ..., ..., ..., ...);

In this approach, all non-English characters, for which the English (United States) has no sorting rule defined, will be sorted by their default sort orders. By doing this, you're guaranteed to retrieve a consistent result.

Important Whenever performing a comparison by using CompareString, the following flags should be used with extra care:
  • NORM_IGNOREKANATYPE. This flag causes corresponding hiragana and katakana characters to compare as equal.
  • NORM_IGNORENONSPACE. This flag causes combined characters (diacritics) to compare as equal to noncombined characters. For example, in Japanese [image] would be equal to [image]. For Latin scripts, "ö" would be equal to "o."

String Sorting

Glossary
  • Alphanumeric: Consisting of either letters or numbers, or both.

You can take advantage of the support for sorting that the operating system provides whenever you represent your data in a list-view control that has the sorting attribute set. Edit controls will sort items by following the linguistic sorting rules specific to the currently selected user locale. But if you want to do your own sorting, the real solution depends on your needs.

If you are sorting only a small number of elements, the easiest and quickest solution is to use the string comparison techniques explained previously, and then to sort elements based on the result of those comparisons. However, this technique does not apply if you are sorting a large number of elements. Because CompareString 'salgorithm is complex, it is faster to sort long lists of words by retrieving sort keys.

Your advanced sorting algorithm should follow this general model:

  1. Generate a sort key for each new element.
  2. Compare strings by comparing the value of their sort keys using memcmp.
  3. Store sort key values along with your string to avoid extra processing time to re-generate them each time they are needed.

A sort key is a composite representation of a sort element. (See Figure 4-21.) Each component of the sort key has a different weight, depending on the sort rules associated with the locale. Elements are sorted based on script (that is, all Greek characters sort together, all Cyrillic characters sort together, and so on); alphanumeric and symbolic characters; diacritics; case; and other special rules, such as two characters sorting as a single character (for example, the Spanish "CH"). The predefined constant SORT_DEFAULT refers to whichever sort algorithm associated with a given language ID is commonly preferred.

Figure 4-21 The Win32 sort key for the German word öde, which means wasteland.
Figure 4-21 - The Win32 sort key for the German word "öde," which means "wasteland."

The first three values in the sort key represent the Unicode sort weights2 for each character in the string. The next three values represent the diacritic weights, and the final three values represent the case weights. Separator values fall between sections. The sort key ends with a null terminator.

The code example that follows contains a search-engine procedure that maps lexical tokens into sort keys. You'll notice that the search engine customizes each sort key by prefixing it with an application-defined flag to ensure that words will sort first, followed by numbers and punctuation. (The system's default behavior is to sort punctuation first, followed by numbers and letters.)

// sort key generation from Unicode
BYTE* pbSource;      // multibyte string buffer
int cbSource;        // size of pbSource buffer

int LCSortKey(
   LCID lcid,        // locale ID
   LPCTSTR pwSource, // address of source string
   int cwSource,     // number of characters in source string
   LPTSTR pwDest,    // address of destination buffer
   int cwDest)       // size of destination buffer
{
   int nRet;         // size of sort key in words

   nRet = LCMapString(lcid, LCMAP_SORTKEY, pwSource, cwSource,
             pwDest+1, (cwDest-1)<<1) >> 1;

   if (nRet)
   {
      nRet++;

      if (cwDest && pwDest) // Set a sort key's prefix so tokens
                            //    group first by alphabetics, then
                            //    by numerics, then by punctuation.
      {
         BYTE bCharType = char_types(*pwSource);
         *pwDest = ~(bCharType & WORD_TYPE);
      }
   }

return nRet;
}

Once the search engine has a sorted list of words, it can quickly look for matching strings using a binary search algorithm. Using sort keys is very convenient because they automatically handle language-specific and locale-specific sorting preferences. One example includes matching ligatures. (In English, "Æ" is equivalent to "AE," but this is not true in Danish.) Another example involves distinguishing between single letters and two-character combinations. (In some Spanish locales, "l" and the combination "ll" are distinct letters.)

Some features of linguistic sorting are:

  • A language's writing system will determine what influences the sort order of the language. For example, a sort order for Russian would be based on Cyrillic letters and possibly diacritics, but a sort order for Japanese might be based on the number of strokes it takes to draw a character.
  • Linguistic sort orders are different than the Unicode code point order.
  • Languages that use the same script often have different linguistic sort orders.
  • A sorting element (such as a character) can be the combination of more than one Unicode code point. For example, U+0B95 + U+0BCD + U+0BB7 = Tamil Ksha, a single sorting element in Tamil.

The best thing to remember is that by using Windows 2000 and Windows XP system support, all of these sorting issues are automatically handled for you.

String Comparison and Sorting in the .NET Framework

Glossary
  • Invariant culture: Whereas the neutral culture is associated with a language, but not with any particular country or region, the invariant culture is neither associated with a language nor with a particular country or region. It can be used in almost any method in the Globalization namespace that requires a culture. The invariant culture must be used only by processes that require culture- and language-independent results, such as system services; otherwise, it produces results that might be linguistically incorrect or culturally inappropriate.

In the following sections, you'll see which methods and properties to use for string comparison. You'll also see what to do when a culture supports more than one sort order, how to use the Array class for string sorting, and how to use sort keys.

String Comparison

The CompareInfo class provides a set of methods you can use to perform culture-sensitive string comparisons. The CultureInfo.CompareInfo property, which is an instance of the CultureInfo class, defines how to compare and sort strings for a specific culture. The String.Compare method uses the information in the CultureInfo.CompareInfo property to compare strings. This method returns a negative integer if string1 is less than string2, zero (0) if string1 and string2 are equal, and a positive integer if string1 is greater than string2.

The following code example illustrates how two strings can be evaluated differently by the String.Compare method depending upon the culture used to perform the comparison. First, the CurrentCulture is set to Danish in Denmark and the strings "Apple" and "Æble" are compared. The Danish language treats the character "Æ" as an individual letter, sorting it after "Z" in the alphabet. Therefore, the string "Æble" is greater than "Apple" for the Danish culture. Next, the CurrentCulture is set to English in the United States and the strings "Apple" and "Æble" are compared again. This time, the string "Æble" is determined to be less than "Apple." The English language treats the character "Æ" as a special symbol, sorting it before the letter "A" in the alphabet.

using System;
using System.Globalization;
using System.Threading;

public class CompareStringSample
{
   public static void Main()
   {
      string str1 = "Apple";
      string str2 = "Æble";

      // Set the CurrentCulture to Danish in Denmark.
      Thread.CurrentThread.CurrentCulture = new CultureInfo("da-DK");

      // Compare the two strings.
      int result1 = String.Compare(str1, str2);
      Console.WriteLine("\nWhen the CurrentCulture is \"da-DK\",\n");
      Console.WriteLine("the result of comparing {0} with {1} is: {2}",
         str1, str2, result1);

      // Set the CurrentCulture to English in the U.S.
      Thread.CurrentThread.CurrentCulture = new CultureInfo("en-US");

      // Compare the two strings.
      int result2 = String.Compare(str1, str2);
      Console.WriteLine(
         "\nWhen the CurrentCulture is \"en-US\",\n");
      Console.WriteLine(
         "the result of comparing {0} with {1} is: {2}",
         str1, str2, result1);
   }
}

If you execute this code, the output appears as follows:

When the CurrentCulture is "da-DK",
the result of comparing Apple with Æble is: -1

When the CurrentCulture is "en-US",
the result of comparing Apple with Æble is: 1

The InvariantCulture is useful for storing data that will not be displayed directly to end users. Storing data in a culture-independent format guarantees a known format that does not change. When users from different cultures access the data, it can be formatted appropriately based on the user. You can initialize CultureInfo with the invariant culture by using an empty string ("") or by using CultureInfo.InvariantCulture as shown:

// The following lines are equivalent.
CultureInfo Invc = New CultureInfo("");
CultureInfo Invc = CultureInfo.InvariantCulture;

Alternate Sort Orders

Glossary
  • Stroke count: The number of strokes it takes to draw an ideographic character.

Some cultures support more than one sort order. For example, the culture "zh-CN" (Chinese in China) supports a sort by pronunciation (default) and a sort by stroke count. When you create a CultureInfo object using a culture name, such as "zh-CN," the default sort order is used. To specify the alternate sort order, create a CultureInfo object using the LCID for the alternate sort order. Then, obtain a CompareInfo object to use in string comparisons from CultureInfo.CompareInfo. As an alternative, you can create a CompareInfo object directlyby using the Com-pareInfo.GetCompareInfo method (Int32), specifying the LCID for the alternate sort order.

Table 4-6 lists the cultures that support alternate sort orders and the LCIDs for the default and alternate sort orders.

Table 4-6 Cultures with alternate sort orders and their corresponding LCIDs.

Culture Name | Language- Country/Region | Default Sort Name and LCID | Alternate Sort Name and LCID
es-ES | Spanish (Spain) | International: 0x00000C0A | Traditional: 0x0000040A
zh-TW | Chinese (Taiwan) | Stroke Count: 0x00000404 | Bopomofo: 0x00030404
zh-CN | Chinese (China) | Pronunciation: 0x00000804 | Stroke Count: 0x00020804
zh-HK | Chinese (Hong Kong SAR) | Stroke Count: 0x00000c04 | Stroke Count: 0x00020c04
zh-SG | Chinese (Singapore) | Pronunciation:0x00001004 | Stroke Count:0x00021004
zh-MO | Chinese (Macau SAR) | Pronunciation: 0x00001404 | Stroke Count:0x00021404
de-DE | German (Germany) | Dictionary:0x00000407 | Phone Book Sort DIN: 0x00010407
hu-HU | Hungarian (Hungary) | Default:0x0000040e | Technical Sort: 0x0001040e
ka-GE | Georgian (Georgia) | Traditional:0x00000437 | Modern Sort:0x00010437

String Sorting

The Array class provides an overloaded Array.Sort method that allows you to sort arrays based on the CultureInfo.CurrentCulture property. In the following example, an array of three strings is created. First, the CurrentCulture is set to "en-US" and the Array.Sort method is called. The resulting sort order is based on sorting conventions for the "en-US" culture. Next, the CurrentCulture is set to "da-DK" and the Array.Sort method is called again. Notice how the resulting sort order differs from the "en-US" results because the sorting conventions for the "da-DK" culture are used.

using System;
using System.Threading;
using System.Globalization;

public class ArraySort
{
   public static void Main(String[] args)
   {
      String str1 = "Apple";
      String str2 = "Æble";
      String str3 = "Zebra";

      // Create and initialize a new Array instance to store
      //    the strings.
      Array stringArray = Array.CreateInstance( typeof(String), 3 );
      stringArray.SetValue(str1, 0 );
      stringArray.SetValue(str2, 1 );
      stringArray.SetValue(str3, 2 );

      // Display the values of the Array.
      Console.WriteLine( "\nThe Array contains the following strings:");
      PrintIndexAndValues(stringArray);

      // Set the CurrentCulture to "en-US".
      Thread.CurrentThread.CurrentCulture = new CultureInfo("enUS");

      // Sort the values of the Array.
      Array.Sort(stringArray);

      // Display the values of the Array.
      Console.WriteLine(
         "\nAfter sorting for the culture \"en-US\":" );
      PrintIndexAndValues(stringArray);

      // Set the CurrentCulture to "da-DK".
      Thread.CurrentThread.CurrentCulture = new CultureInfo("daDK");

      // Sort the values of the Array.
      Array.Sort(stringArray);

      // Display the values of the Array.
      Console.WriteLine(
         "\nAfter sorting for the culture \"da-DK\":" );
      PrintIndexAndValues(stringArray);
   }

public static void PrintIndexAndValues(Array myArray)
   {
    for ( int i = myArray.GetLowerBound(0);
        i <= myArray.GetUpperBound(0); i++ )
      Console.WriteLine( "\t[{0}]:\t{1}", i, myArray.GetValue( i ) );
   }
}

This code produces the following output:

The Array initially contains the following strings:

   [0]:   Apple
   [1]:   Æble
   [2]:   Zebra

After sorting for the culture "en-US":

   [0]:   Æble
   [1]:   Apple
   [2]:   Zebra

After sorting for the culture "da-DK":

   [0]:   Apple
   [1]:   Zebra
   [2]:   Æble

Using Sort Keys

Once again, sort keys are used to support sorting methods that vary from one culture to another. According to sort-key methodology, each character in a string is given several categories of sort weights, including alphabetic, case, and diacritic weights. A sort key serves as the repository of these weights for a particular string. For example, a sort key might contain a string of alphabetic weights, followed by a string of case weights, and so on.

In the .NET Framework, the SortKey class maps strings to their sort keys and vice versa. You can use the CompareInfo.GetSortKey method to create a sort key for a string that you specify. The resulting sort key for a specified string is a sequence of bytes that can differ depending upon the CurrentCulture and the CompareOptions that you specify. For example, if you specify IgnoreCase when creating a sort key, a string comparison operation using the sort key will ignore case.

After you create a sort key for a string, you can pass it as a parameter to methods provided by the SortKey class. The SortKey.Compare method allows you to compare sort keys. Because SortKey.Compare performs a simple byte-by-bytecomparison, it is much faster than using String.Compare. In applications that are sorting-intensive, you can improve performance by generating and storing sort keys for all the strings that the application uses. When a sort or comparison operation is required, you can use the sort keys rather than the strings.

The following code example creates sort keys for two strings when the CurrentCulture is set to "da-DK". It compares the two strings using the Sort-Key.Compare method and displays the results. The SortKey.Compare method returns a negative integer if string1 is less than string2, zero (0) if string1 and string2 are equal, and a positive integer if string1 is greater than string2. Next, the CurrentCulture is set to "en-US" culture, and sort keys are created for the same strings. The sort keys for the strings are compared and the results are displayed. Notice that the sort results differ based upon the CurrentCulture. Although the results of the following code example are identical to the results of comparing these strings in the String.Compare example given earlier, the SortKey.Compare method is faster than the String.Compare method.

using System;
using System.Threading;
using System.Globalization;

public class SortKeySample
{
   public static void Main(String[] args)
   {
      String str1 = "Apple";
      String str2 = "Æble";

      // Set the CurrentCulture to "da-DK".
      CultureInfo dk = new CultureInfo("da-DK");
      Thread.CurrentThread.CurrentCulture = dk;

      // Create a culturally sensitive sort key for str1.
      SortKey sc1 = dk.CompareInfo.GetSortKey(str1);

      // Create a culturally sensitive sort key for str2.
      SortKey sc2 = dk.CompareInfo.GetSortKey(str2);

      // Compare the two sort keys and display the results.
      int result1 = SortKey.Compare(sc1, sc2);
      Console.WriteLine(
         "\nWhen the CurrentCulture is \"da-DK\",\n");

      Console.WriteLine("\the result of comparing {0} with {1} is: {2}",
         str1, str2, result1);

      // Set the CurrentCulture to "en-US".
      CultureInfo enus = new CultureInfo("en-US");
      Thread.CurrentThread.CurrentCulture = enus ;

      // Create a culturally sensitive sort key for str1.
      SortKey sc3 = enus.CompareInfo.GetSortKey(str1);
      // Create a culturally sensitive sort key for str1.
      SortKey sc4 = enus.CompareInfo.GetSortKey(str2);
      // Compare the two sort keys and display the results.
      int result2 = SortKey.Compare(sc3, sc4);
      Console.WriteLine(
         "\nWhen the CurrentCulture is \"en-US\",\n");
      Console.WriteLine(
         "\the result of comparing {0} with {1} is: {2}",
         str1, str2, result2);
   }
}

This code produces the following output:

When the CurrentCulture
 is "da-DK",
the result of comparing Apple with Æble is: -1
When the CurrentCulture
 is "en-US",
the result of comparing Apple with Æble is: 1

In addition to dealing with varying sort orders, representing numbers according to locale conventions is another challenge that you'll encounter. The sections that follow show how you can best meet this challenge.


Number Formatting

When dealing with numeric values, there are six major items to pay attention to:

The character used as the thousands separator. In the United States, this character is a comma (,). In Germany, it is a period (.). Thus one thousand and twenty-five is displayed as 1,025 in the United States and 1.025 in Germany. In Sweden, the thousands separator is a space.

The character used as the decimal separator. In the United States, this character is a period (.). In Germany, it is a comma (,). Thus one thousand twenty-five and seven tenths is displayed as 1,025.7 in the United States and 1.025,7 in Germany.

The way negative numbers are displayed. The negative sign can be used at the beginning of the number, but it can also be used at the end of the number. Alternatively, the number can be displayed with parentheses around it. Thus a negative five hundred and twenty-seven could be displayed as:

    -527

    527-

    (527)

The shape of the numbers, or whether they have a one-to-one correspondence to the decimal set of digits. Numbersmight, of course, be shaped differently from one locale to another. Also, numbers in some locales might not correspond directly to the digits (0-9) used in, say, the United States. (See Table 4-7.)

Table 4-7 Digits used for English (United States), Arabic, and Japanese locales.

Locale | Digits Used
English (United States) | 0 1 2 3 4 5 6 7 8 9
Arabic |
Japanese |

Note how Japanese has one more digit than Arabic or English (United States). This represents the number 10.

Digit grouping. This refers to the number of digits contained between each separator for all digit groups that appear to the left of the decimal separator. For example, the 3-digit group is used for most cultures, such as for English (United States): 123,456,789.00. However, notice that Hindi uses a 2-digit grouping, except for the 3-digit grouping for denoting hundreds: 12,34,56,789.00

The placement of the percent sign (%). It can be written several ways: 98%, 98 %, 98 pct, %98. Thus you should never assume that you can hard-code the percent sign.

The user can define preferred number-formatting parameters by making selections from the Numbers tab of the Customize Regional Options property sheet, within the Regional And Language Options property sheet. (See Figure 4-22.)

Figure 4-22 Selecting the preferred number formatting.
Figure 4-22 - Selecting the preferred number formatting.

Number Formatting in Win32

The easiest way to format numbers in a way that's locale-aware is to use the GetNumberFormat API. This API customizes the format of a number string for a specified locale. The following shows how the GetNumberFormat API is used to format a given number string for the current user locale (using the default settings for number formatting that the user has defined):

GetNumberFormat(LOCALE_USER_DEFAULT, // locale (current user locale)
   0,                                // options
   TEXT("1234567890.12345"),         // input string (see MSDN for legal
                                     //    chars)
   NULL,                             // formatting information
   g_szTemp,                         // formatted string buffer
   MAX_STR);                         // size of buffer

For more advanced number formatting in the Win32 programming model, the GetLocaleInfo API can be used to retrieve most of the number-formatting parameters (such as thousands separators, decimal separators, and negative numbers). Here are the appropriate flags to use as LCType in your calls to GetLocaleInfo, in order to retrieve each one of the number-formatting parameters.

Thousands separator. LCType flag set to LOCALE_STHOUSAND

Decimal separator. LCType flag set to LOCALE_SDECIMAL

Negative numbers. LCType flag set to LOCALE_INEGNUMBER

Possible return values are shown in Table 4-8.

Table 4-8 Return values of various LCType flags.

Value | Format (meaning)
0 | Left parenthesis, number, right parenthesis. Example: (1.1)
1 | Negative sign, number. Example: -1.1
2 | Negative sign, space, number. Example: - 1.1
3 | Number, negative sign. Example: 1.1-
4 | Number, space, negative sign. Example: 1.1 -

Native digits. LCType flag set to LOCALE_SNATIVEDIGITS. Here is a code sample that deals with native digits:

GetLocaleInfo(LOCALE_USER_DEFAULT,  // LCID (current user locale)
   LOCALE_SNATIVEDIGITS,            // information type (native digits)
   &g_szTemp,                       // returned value
   sizeof (g_szTemp));              // size of buffer

This code would produce the following result on English (US) and Farsi locales, respectively. (See Figure 4-23.)

Figure 4-23 Native digits for English (US) and Farsi.
Figure 4-23 - Native digits for English (US) and Farsi.

Digit Substitution. Digit substitution defines which set of digits (with its associated shapes) should be used for presenting numbers. This information can be retrieved with the LCType flag set to LOCALE_IDIGITSUBSTITUTION.

Table 4-9 shows the possible return values.

Table 4-9 Return values for LOCALE_IDIGITSUBSTITUTION.

Value | Meaning
0 | Context-the shape depends on the previous text in the same output.
1 | None/Arabic-gives full Unicode compatibility.
2 | Native-national shapes determined by LOCALE_SNATIVEDIGITS.

Digit grouping. LCType flag set to LOCALE_SGROUPING

Number Formatting in Web Pages

In "Retrieving the Browser Language Setting" earlier in this chapter, you saw how to retrieve the current browser locale on the client side and how to set the global locale of your context or session to this value. Once the appropriate locale has been set, you can easily format numbers by using FormatNumber, a locale-aware function.

With this function, formatting 123456.789 would become:

For English (United States): 123,456.789

For Arabic (Egypt): , [image].

Obviously the scripting technology does not offer the same flexibility to manipulate number formatting as NLS APIs do in the case of Win32 programming. However, the FormatNumber function enables you to display numbers according to the cultural format that the user prefers.

Number Formatting in the .NET Framework

The NumberFormatInfo class defines how currency, decimal separators, and other numeric symbols are formatted and displayed based on culture. For example, the decimal number 10000.50 is formatted as 10,000.50 for the culture "en-US" and 10.000,50 for the culture "de-DE." An instance of NumberFormatInfo can be created for a specific cultureor the invariant culture, but not for a neutral culture. A neutral culture does not provide enough information to display the correct numeric format. Table 4-10 lists the standard format characters for each standard formatting pattern and the associated NumberFormatInfo property that can be set to modify this pattern.

Table 4-10 Standard format characters for basic formatting patterns and the associated NumberFormatInfo property used to modify these patterns.

Format Character | Description and Associated Properties
c, C | Currency format. CurrencyNegativePattern, CurrencyPositivePattern, CurrencySymbol, CurrencyGroupSizes, CurrencyGroupSeparator, CurrencyDecimalDigits, CurrencyDecimalSeparator
d, D | Decimal format
e, E | Scientific (exponential) format
f, F | Fixed-point format
g, G | General format
n, N | Number format. NumberNegativePattern, NumberGroupSizes, NumberGroupSeparator, NumberDecimalDigits, NumberDecimalSeparator
r, R | Roundtrip format, which ensures that numbers converted to strings will have the same value when they are converted back to numbers
x, X | Hexadecimal format

The following code example displays an integer using the NumberFormatInfo standard currency format ("c") for the specified CurrentCulture.

using System;
using System.Globalization;

public class TestClass
{
   public static void Main()
   {
      int i = 100;

      // Create a CultureInfo object for English in Belize.
      CultureInfo bz = new CultureInfo("en-BZ");

      // Display i formatted as currency for bz.
      Console.WriteLine(i.ToString("c", bz));

      // Create a CultureInfo object for English in the U.S.
      CultureInfo us = new CultureInfo("en-US");
      // Display i formatted as currency for us.
      Console.WriteLine(i.ToString("c", us));

      // Create a CultureInfo object for Danish in Denmark.
      CultureInfo dk = new CultureInfo("da-DK");

      // Display i formatted as currency for dk.
      Console.WriteLine(i.ToString("c", dk));
   }
}

This code produces the following output:

BZ$100.00
$100.00
kr100,00

As when dealing with numbers, formatting conventions for addresses vary widely from one country to another. There are also discrepancies in the actual terms used when representing addresses that you will need to handle.


Addresses

One of the most nonstandard formatted itemsyou'll need to deal with in globalization is address formats. Thus input fields and the routines that process address information should be able to handle various formats. For instance, a very common mistake (such as in Web forms) is to insist that the user entersomething in a field labeled State (or Province for Canadians). While this makes sense to people located in the United States and Canada, it confuses those from other parts of the world, since most don't have a "State" in their addresses.

You must also be flexible when performing validity checks of the dataentered by the user. For example, don't assume that theZIP code (or postal code, as it is referred to in many countries outside the United States) has any particular format or length, or that it comprises only digits.For instance, Canadian postal codes consist of two groups of three characters, such as "M5R 3H5"; a French postal code is a five-digit number, as in 92300. Insome places, people might add a country or region code in front of the postal code (for example, F-92300).

The current implementation of NLS APIs and the .NET Framework do not provide any address formatting information. The best approach is to:

  • Divide the address into multiple fields for street number, building number, country, city, and code.
  • Don't expect that all predefined fields should contain a value (such as the previous example of postal codes).
  • Be flexible for additional data that you might not usually expect in an address (such as a description of how to get there).

(For a list of the preferred address formats for selected countries, see Appendix S, "International Address Formats.")


Paper Size

The paper sizes in the United States and Canada (such as letter, legal, and so on) do not satisfy the needs of all users in the world market. For example, most countries in Europe and Asia use a slightly larger standard known as "A4" (297 x 210 mm) that is slightly longer and narrower than the U.S. letter size (279 x 216 mm). Thus if your application needs to print, you should allow the default paper size to be configurable. Win32 NLS APIs can help you detect the default paper size associated with a given locale. This information can be obtained from the GetLocaleInfo API with the LCType flag set to LOCALE_IPAPERSIZE. The returned value is one of the following. (See Table 4-11.)

Table 4-11 Returned values for LOCALE_IPAPERSIZE.

Value | Meaning
1 | U.S. letter
5 | U.S. legal
8 | A3
9 | A4


Telephone Numbers

Like addresses, the format for telephone numbers around the world varies significantly. Once again, input fields and the routines that process information dealing with telephone numbers should be able to handle the variety of formats.

Table 4-12 displays different formats used throughout the world.

Table 4-12 Formats for telephone numbers for selected countries.

Country | Format
China | 1234 5678
France | 01-23-45-67-89
Poland | (12) 345.67.89
Singapore | 123 4567
Thailand | (01) 234-5678 or (012) 34-5678
United Kingdom | 0123 456 7890 or 01234 567890
United States | (123) 456 7890

Notice that there are different separators such as hyphens (-), periods (.), and spaces, different groupings (two, three, four, five, and six digits per group), and different numbers of total digits used (7-11). Also, the examples just given didn't include country codes, which could be anything from one to three digits.

The ITU-T standard E.164, defined by the Comité Consultatif International Téléphonique et Télégraphique (CCITT), states that the maximum number of digits is 15, but this doesn't include space for things like:

  • "Exit" numbers for corporate telephone systems (such as, in the United States, dialing a "9" in order to place a call outside the place of business)
  • Long-distance access codes
  • Passwords
  • Credit card numbers
  • Extensions

Thus when designing and coding for display and storage of telephone numbers, do not assume one given format, but leave it very flexible. The current implementation of NLS APIs and the .NET Framework do not provide any telephone-number formatting information. Knowing this, you should:

  • Keep country codes, area codes, and phone numbers in separate fields.
  • Not expect a fixed number of digits for an area code or phone number. (A phone number can be anything between 4 to 11 digits.)
  • Expect different separators-hyphens (-), periods (.), and commas (,)-between numbers. (You will need to do your own parsing for these separators.)


Units of Measure

Throughout the world things are measured using different units and scales. The most popular method of measurement used is the metric system (meters, liters, grams, and so on). (The most current version of the metric system is known as "Système Internationale d'Unités," or "S.I.") The United States, on the other hand, uses the U.S. Customary System3 (feet, inches, pounds, and so on). The kind of measurements referred to are for things like the following:

  • Lengths
  • Weights
  • Area
  • Volume
  • Temperatures
  • Paper sizes
  • Angle notation

Thus you need to make sure that if you deal with measurements, you can display them in different measurement systems. Also, make sure it is clear to the user which system is currently being displayed. Recall the unfortunate incident in September 1999 involving a Mars probe that strayed off course, partly because of controllers' faulty data conversions from U.S. Customary to metric units of measurement.

Units of Measure in Win32

Although you need to do your own conversions between the metric and U.S. systems, Win32 NLS APIs can help you detect which system of measurement is used for a given locale. The system of measurement can be obtained from the GetLocaleInfo API with the LCType flag set to LOCALE_IMEASURE. The returned value is 0 if the metric system is used, and 1 if the U.S. Customary System is used. The maximum number of characters allowed for this string is two.

Units of Measure in the .NET Framework

The RegionInfo class from the System.Globalization namespace contains information about the country or region. In contrast to CultureInfo, RegionInfo does not represent preferences of the user and does not depend on the user's language or culture. A good example of this category of information is the IsMetric property of RegionInfo that gets a Boolean value indicating whether the country or region uses the metric system for measurements.

Until now, this chapter has dealt with locale awareness in terms of Win32 applications, Web pages, and the .NET Framework. The final major area to consider is how to handle language-specific operations with console applications. The final sections of this chapter, therefore, will concentrate on this task.


Local Awareness in Console

Locale Awareness in Console

Language-specific operations of console applications can follow the locale settings of the system. This will guarantee locale-authentic formatting of numbers, date, time, currency values, and collation (sorting) operations. Locale support is available for text-mode applications through both C run-time (CRT) and Win32. However, these mechanisms should never be mixed. For new Windows applications, Win32 mechanisms are preferred over those in CRT with regard to world-ready code.

C Run Time

CRT locale support is built around the (_w)setlocale(category, locale) call, discussed in Chapter 3. A call to this function defines the results of all subsequent CRT-based locale-sensitive operations, not only the character encoding. The category argument defines scope of environment changes after setlocale is called.

In order to set the rules for formatting locale-sensitive data in accordance with the user locale, the following calls can be executed:

_wsetlocale (LC_COLLATE, L(".OCP") );  // sets the sort order
_wsetlocale (LC_MONETARY, L(".OCP") ); // sets the currency formatting
                                       //    rules
_wsetlocale (LC_NUMERIC, L(".OCP") );  // sets the formatting of
                                       //    numerals
_wsetlocale (LC_TIME, L(".OCP") );     // defines the date/time
                                       //    formatting

As mentioned in Chapter 3, ".OCP" and ".ACP" parameters always refer to the settings of the user locale, not the system locale. While selecting this locale for LC_CTYPE or LC_ALL is not a good choice, all other categories should be set to match the user locale, unless your console must be explicitly independent of the user's settings.

Win32 and .NET Framework

Console applications based on the Win32 API and .NET Framework are no different from other Win32 or .NET applications in their ability to access the Win32 NLS API or the System.Globalization namespace. Using them, however, might cause some display problems because of the special rules that apply to console input/output (I/O), as described in Chapter 3. The user locale or CurrentCulture might require the date and time to be formatted using characters in a range unsupported by the console. For example, you cannot write out dates in a long Hebrew format or times with Arabic A.M./P.M. symbols. Some special design decisions must be made to avoid displaying culturally correct yet unreadable data. However, you should avoid taking the easy route of hard-coding the format for your output.


Summary

Universal encoding is only one component of globalized software. A second component involves locale awareness. That is, the application must observe the language rules and cultural conventions that the user has selected. Representing such things as the date, time, currency symbols, numerals, addresses, andtelephone numbers in a format that matches the user's language, culture, and country standards is an essential task.

Locale variables define various system components that deal with international support. However, of all the variables available on Windows, only the user locale, the location, the browser language setting, and the thread locale deal with locale awareness in applications.

Within the Win32 programming environment, NLS APIs provide the right tools to eliminate any locale-specific assumptions from your application, allowing you to represent data in the formatting that corresponds to the currently selected user locale identified by a locale ID (LCID). The .NET Framework took a new approach to the definition of locale identification and readjusted the misnaming introduced by NLS APIs with its introduction of culture information, represented by the CultureInfo class from theSystem.Globalization namespace. This class contains culture-specific information, such as the language, country or region, calendar, and cultural conventions. This class also provides the information required for performing culture-specific operations, such as casing, formatting dates and numbers, and comparing strings.

Since all APIs and functions that allow locale-specific formatting take as an argument the locale ID or the culture name for which this formatting should be performed, the first step in writing locale-aware software is to retrieve the proper locale.

String sorting and comparison is another important part of the process. Even within languages based on the Latin script, there are different composition and sorting rules. Thus, do not rely on code points to do proper sorting and string comparison. The Win32 functions lstrcmp and lstrcmpi compare two character strings in a case-sensitive or case-insensitive manner, respectively, following the sorting rules and standards that are defined for the currently selected user locale. In the .NET Framework, the CompareInfo class provides a set of methods you can use to perform culture-sensitive string comparisons.

Finally, language-specific operations of console applications can follow the locale settings of the system. This will guarantee locale-authentic formatting of numbers, date, time, currency values, and collation (sorting) operations. Locale support is available for character-mode applications through both CRT and Win32. However, these mechanisms should never be mixed. For new Windows applications, Win32 mechanisms are preferred over those in CRT with regard to world-ready code.


Chapter 5: Text Input, Output, and Display
Glossary
  • Keyboard layout: A standard arrangement of characters on a keyboard that defines which keys produce particular characters or scan codes.
  • Scan code: The value sent from the keyboard to the keyboard driver that represents which key was pressed.

On the surface, the input, output, and display of data is a simple process-the user enters data with a keyboard (or perhaps with a voice-recognition or handwriting-recognition engine), and the program displays that data using a selected font. This assessment of the process, however, assumes that each user is monolingual. What if a user tries to enter data in several different languages? In this case, the process becomes much more complicated. Different languages have different keyboard layouts, and the characters of each language might fall into different character sets that require separate fonts. How can Microsoft Windows-based applications juggle multiple keyboards, character sets, and fonts? With all these challenges in mind, Microsoft Windows 2000 and Microsoft Windows XP support standard mechanisms for multilingual data input, output, and display.

Creating an application that supports multilingual data in Windows 2000 and Windows XP involves several steps. The first step is to fully enable your code to use Unicode standards. (See Chapter 3, "Unicode." ) Doing so will allow you to encode all supported scripts in a uniform manner and to empower users and programmers to handle multilingual data. The second step is to make sure that your application is locale- and culture-aware by taking advantage of Windows globalization services (including National Language Support application programming interfaces [NLS APIs]) in traditional Microsoft Win32 coding and the Microsoft .NET Framework for managed code). These services will allow you to format locale- or culture-sensitive data according to user preference. (See Chapter 4, "Locale and Cultural Awareness." )

The next tasks when globalizing your application, all of which are discussed in this chapter, are to:

  • Enable your application to deal with different input languages and methods. Included are traditional methods such as different keyboard layouts and Input Method Editors (IMEs), as well as new input services such as voice-recognition engines or handwriting-recognition engines. (For more information on how to deal with different input languages, see "Techniques for Handling Input Languages" later in this chapter.)
  • Properly display all supported scripts, in accordance with the linguistic characteristics associated with them. These scripts include complex scripts that require special handling in terms of layout and character shaping. (For more information on complex scripts, see "Complex Scripts" later in this chapter.)
  • Understand different options for displaying text by leveraging system support for multilingual computing. (For more information on displaying text, see "Options for Displaying Text" later in this chapter.)
  • Learn how you should handle fonts in a multilingual context. (For more information on fonts, see "Fonts" later in this chapter.)

To begin, this chapter will examine the role that input languages and methods play in multilingual computing.


Input Languages

Glossary
  • Input method: Any method used to enter text. These methods include different keyboard layouts and IMEs, as well as newer input services such as voice-recognition engines or handwriting-recognition engines.

Before diving into the technical aspects of how to support input languages, this chapter will first show you how a typical user interacts with these languages. Because Windows 2000 and Windows XP allow the user to enter multiple languages using a variety of input methods, the system needs to know which in put method should be active for a particular language. These associations are called "installed language and method pairs," or "input languages" (called "input locales" in Windows 2000). During installation, the default input language for the language version of the operating system, along with English, is installed for each user. The user can then define the list of input languages to be made available for his or her own account and usage. For example, on the same machine, one user can have an English keyboard layout and a Japanese IME installed, and another user can have both French and Arabic keyboard layouts installed. This customization is done by adding or removing input languages and using them on the fly from the Regional And Language Options property sheet, provided that the language support of the target language has already been installed. (See Figure 5-1.)

Figure 5-1 Each user can add and remove input languages from the Languages tab of the Regional And Language Options property sheet.
Figure 5-1 - Each user can add and remove input languages from the Languages tab of the Regional And Language Options property sheet.

The default input language is the input language that is active when a new application thread is started. Switching to a different input language is done on a per-thread basis; you can have two different input languages in two different applications. The taskbar indicates which input language is currently active. For example, in Figure 5-2, English is the input language that is currently active. When the user clicks the language indicator in the taskbar-each language is represented by its two-letter abbreviation-Windows 2000 and Windows XP present a list of alternatives such as Japanese, French (Canada), and so on. (For an extensive list of locales and their associated valid input languages for Windows XP, see Appendix P, "List of Keyboards and IMEs Supported by Microsoft Windows 2000 and Microsoft Windows XP.")

Figure 5-2 List of available input languages, with English being the one that is currently active for this particular user.
Figure 5-2 - List of available input languages, with English being the one that is currently active for this particular user.

The shortcut keys iterate through the list of installed language and method pairs in the order in which they were added via the Regional And Language Options property sheet. If the user has selected Left Alt+Shift in the Advanced Key Settings dialog box, Left Alt+Shift will allow the user to toggle between different installed input languages. (See Figure 5-3.)

Figure 5-3 Switching between various input languages in Windows XP.
Figure 5-3 - Switching between various input languages in Windows XP.

Having gained an understanding of how the user can customize a list of input languages and switch from one input language to another, you'll now see the most efficient ways to work with input languages from a developer's standpoint. Taking advantage of system support will go a long way toward making your job easier.

Techniques for Handling Input Languages

The Microsoft Developer Network (MSDN) documentation (found at http://msdn.microsoft.com) and programming APIs represent input languages with a variable type called "input locale identifier," formally known in older documentation as "Handle to the Keyboard Layout" (HKL) and still used as the type identifier. HKL is an archaic name from a time when the only input was from a keyboard. The input locale identifier name is a 32-bit value composed of the hexadecimal value of the language identifier (low WORD) and a device identifier (high WORD). (See Figure 5-4.) For example, U.S. English has a language identifier of 0x0409, so the primary U.S. English layout is named "00000409." Variants of the U.S. English layout (such as the Dvorak layout) are named "00010409," "00020409," and so on. The device identifier is not limited to keyboards and IMEs; data can now be entered by more sophisticated mechanisms such as voice- and text- recognition engines. For instance, Microsoft Windows Text Services Framework (TSF)-a system service available on Windows XP-enables advanced, source-independent text input. (For more information on TSF, see Chapter 23, "Microsoft Windows Text Services Framework [TSF].")

Figure 5-4 The HKL variable, which represents input languages.
Figure 5-4 - The HKL variable, which represents input languages.

The easiest way to handle input languages is to use the standard controls that the operating system provides whenever you are expecting user input. For example, by using Unicode edit controls or rich edit controls, you enable your application to handle multilingual text input. The operating system automatically handles input languages in a way that is transparent to your application. For instance, the GlobalDev application is a property sheet with a tab called "Text APIs." Text APIs uses a standard multiline edit control, which eliminates the hassle of dealing with input languages. (You can find the GlobalDev application and the Text APIs page in the Samples subdirectory on the companion CD.)

Advanced applications (such as a text editor) that need to have full control over how input languages are handled should monitor-and should be able to respond to-the user's changes. When a user selects an input language by clicking on the language indicator of the taskbar or by pressing Left Alt+Shift, the input language is not automatically changed-either action generates a request that the active application must accept or reject. In response to the hot-key combination or the mouse click on the language indicator of the taskbar, the system sends a WM_INPUTLANGCHANGEREQUEST message to the window of focus, as Figure 5-5 illustrates. If the application accepts the message and passes it to DefWindowProc, the system initiates switching the input language, sending a WM_INPUTLANGCHANGE message. The process is slightly different when the input method is a part of the Text Services Framework (TSF), in which case only a WM_INPUTLANGCHANGE is sent. When the system successfully completes the change, it generates a WM_INPUTLANGCHANGE message. The lParam variable of the WM_INPUTLANGCHANGE message contains the input locale identifier (that is, the HKL) of the new input language.

Figure 5-5 WM_INPUTLANGCHANGEREQUEST and WM_INPUTLANGCHANGE message propagation flowchart.
Figure 5-5 - WM_INPUTLANGCHANGEREQUEST and WM_INPUTLANGCHANGE message propagation flowchart.

An application that does not support multiple languages will reject the WM_INPUTLANGCHANGEREQUEST message. It might reject any or all WM_INPUTLANGCHANGEREQUEST messages, or it might perform a couple of tests first. For example, the wParam variable of this message is a Boolean value-bLangInSystemCharset -that indicates whether the requested input language can be represented in the current system locale. Representing input languages is not a worry when dealing with Unicode applications, but non-Unicode applications should, in fact, monitor this value, or they will display the wrong characters.

Similar to the system generating a WM_INPUTLANGCHANGEREQUEST message in response to a user request, applications can also initiate input language changes by calling the ActivateKeyboardLayout API. This allows a user who is editing a document containing Latin and Greek text to automatically activate the Greek input method when moving the insertion point from the Latin text to the Greek text. (See Figure 5-6.) Likewise, when this user moves the insertion point back to the Latin text, the application will activate the default Latin-based input method.

Figure 5-6 When the cursor is positioned in a Greek text stream, the active keyboard layout should switch to Greek.
Figure 5-6 - When the cursor is positioned in a Greek text stream, the active keyboard layout should switch to Greek.

Other Win32 APIs that handle input methods are shown in Table 5-1.

Table 5-1 Win32 APIs that handle input methods.

Keyboard-Related API Function | Description
GetKeyboardLayout | Returns the active installed language and method pair
GetKeyboardLayoutList | Returns a list of installed language and method pairs
GetKeyboardLayoutName | Returns the name of the active input method
LoadKeyboardLayout | Loads a new input method into the system
UnloadKeyboardLayout | Unloads an input method; cannot unload the system default
ActivateKeyboardLayout | Changes the active installed language and method pair

When you design functionality to allow the user to switch keyboard layouts, keep in mind that because the letters on keyboards vary from layout to layout, the keys used to generate shortcut-key combinations might also vary. For example, the French keyboard defaults to the AZERTY layout, whereas the English layout follows a QWERTY mapping. Therefore, it is suggested that you use numbers and function keys (F4, F5, and so on) instead of letters in shortcut-key combinations.

In addition to enabling your application to handle varying input languages, you will also need to enable it to support IMEs. (Keep in mind that if you use standard APIs for input, your applications will automatically handle IMEs.) By enabling IME support, you allow the user to enter ideographs, for example, from various East Asian writing systems. The following sections explore what an IME does-with practical examples and technical solutions on the best ways to support IMEs.

Input Method Editors

Glossary
  • Conversion or composition window: The window of an IME that displays text typed by the user, either just the way it is entered or after it is converted to ideographic form.
  • Status window: The window of an IME in which the user can change the IME's conversion mode or input mode.
  • Candidate window: The window of an IME that lists characters the user can choose to replace the text highlighted in the composition window.
  • Input Method Manager (IMM): The module on Windows 2000 and Windows XP that handles communication between IMEs and applications.
  • Dead key: A key that does not produce a character by itself, such as the accent key on the international keyboard. However, when the user types in a character after pressing the accent key, an accented character appears.

IMEs are components that allow the user to enter the thousands of different characters used in East Asian languages using a standard 101-key keyboard. The user composes each character in one of several ways: by radical, by phonetic representation, or by typing in the character's numeric code-page index. IMEs are widely available; Windows 2000 and Windows XP ship with standard IMEs that are based on the most popular input methods used in each target country, and a number of third-party vendors sell IME packages.

An IME consists of an engine that converts keystrokes into phonetic and ideographic characters, plus a dictionary of commonly used ideographic words. As the user enters keystrokes, the IME engine attempts to guess which ideographic character or characters the keystrokes should be converted into. Because many ideographs have identical pronunciation, the IME engine's first guess isn't always correct. When the suggestion is incorrect, the user can choose from a list of homophones; for more advanced IMEs, the homophone that the user selects then becomes the IME engine's first guess the next time around. This process is summarized in Figure 5-7.

Figure 5-7 The process through which an IME engine converts keystrokes into ideographic characters.
Figure 5-7 - The process through which an IME engine converts keystrokes into ideographic characters.

Before examining how the user can enter ideographs using IMEs and how you can add support for IMEs in your application, the next sections will give a quick overview of the linguistic differences among East Asian scripts.

East Asian Writing Systems

Chinese, Japanese, and Korean writing systems all offer some interesting complexities not found in Latin writing systems. To put things in clearer context, it will be useful for you to have an idea of what these complexities entail.

Chinese Three forms of ideographic characters are commonly used today in the world: Traditional Chinese, Simplified Chinese, and kanji (which is used for Japanese). Traditional Chinese characters, which are thousands of years old and have kept their original shapes, generally contain more strokes than other ideographic forms, and are more pictorial. These characters are typically used in Taiwan. Simplified Chinese characters, which are based on Traditional Chinese characters, were developed in mainland China to make reading and writing easier to learn. Although Traditional Chinese and Simplified Chinese share some characters, the simplified characters, of which there are less than 7,000, are composed of fewer strokes and in most cases are distinct from their original counterparts. This is why software products developed for the Chinese-speaking market are usually released in two editions-one for the Traditional Chinese script and one for the Simplified Chinese script.

Japanese Japanese characters are called "kanji." Japanese mixes kanji characters with characters from two syllabaries, collectively called "kana." The two forms of kana are referred to as "hiragana" and "katakana." Hiragana is a cursive script, commonly used in Japanese text to represent ending inflections for verbs and to write native Japanese words that have no kanji equivalent, such as "and," "of," and "to." Katakana is chiefly used to represent words borrowed from other languages. All kana symbols, except for single-vowel characters and the character "n," represent a consonant followed by one of five vowels. Hiragana and katakana both represent the entire Japanese script of sounds.

Korean The Korean written language uses two types of characters: hangul and hanja. A hangul character is a single syllabic character created by combining one or more consonant signs and a vowel sign. There are 24 basic elements (14 consonants and 10 vowels), or phonemes, used to denote these signs; these elements are called "jamos." You can create up to 51 jamos by combining two or more basic elements to form additional vowels or consonants, called "compounds." Compounds and basic elements together comprise 21 vowels (10 basic vowels and 11 compound vowels) and 30 consonants (14 basic consonants and 16 compound consonants). A hangul character (syllabic) consists of an initial consonant, a medial vowel, and sometimes a final consonant. Nineteen of the 30 consonants can be initial consonants. All 21 vowels can be medial vowels, and 27 of the 30 consonants can be final consonants. This means that 11,172 hangul character combinations are possible, though far fewer are actually used. The Korean language also adopted hanja characters from Chinese and uses them for more formal written communication and to represent personal names. Most daily communication is written in hangul.

Ways to Enter Ideographs with an IME

With an IME you don't have to use a localized keyboard to enter ideographic characters. While East Asian keyboards can generate phonetic syllables (such as kana or hangul) directly, the user can represent phonetic syllables using Latin characters. In Japanese, Latin characters that represent kana are called "romaji." Japanese keyboards contain extra keys that allow the user to toggle between entering romaji and entering kana. If you are using a non-Japanese keyboard, you need to type in romaji to generate kana.

The best way to learn how an IME works from the user's perspective is to try using it and to take advantage of the extensive Windows Help files. As a reference, the following sections look at how the Japanese IME that ships with Windows XP works.

The Standard Japanese IME for Windows XP

The Japanese IME for Windows XP, called "Microsoft IME 2002" (see Figure 5-8), has six standard input modes, listed in Table 5-2. Additionally, IME 2002 contains an IME Pad that allows for alternative methods of input, and several other tools for handling both conversion into kanji and voice input. Although you will usually see IME 2002 the way it appears in Figure 5-8, it also has a drop-down menu that lists various input modes. (See Figure 5-9.)

Figure 5-8 The Japanese IME Language bar.
Figure 5-8 - The Japanese IME Language bar.
Table 5-2 The Japanese IME input modes.

Japanese IME Input Mode | IME Toolbar Setting | Key to Convert Text Representation
Full-width hiragana |  | f6
Full-width katakana |  | f7
Full-width alphanumeric |  | f9
Half-width katakana |  | f8
Half-width alphanumeric |  | f8
Direct input |  | Not applicable
Figure 5-9 IME 2002 on Windows XP. The input modes are listed in the drop-down menu. The last input mode, called direct input, turns off the IME, and keystrokes are sent to the application directly without being converted into phonetic syllables.
Figure 5-9 - IME 2002 on Windows XP. The input modes are listed in the drop-down menu. The last input mode, called "direct input," turns off the IME, and keystrokes are sent to the application directly without being converted into phonetic syllables.

Input of Japanese Characters In order to begin entering Japanese characters in an application running on Windows XP, you need to activate the IME by selecting it from the list of input languages. When you activate the IME, the floating Language bar changes to the Japanese IME toolbar as you saw in Figure 5-8. Table 5-3 shows what happens when you enter Japanese characters into an application running on Windows XP.

Table 5-3 Entering Japanese characters in an application running on Windows XP.

Action | Result
Type the letter "k." The IME conversion (or composition) window is represented by a dotted underline. The window might be displayed anywhere on the screen, but in most applications it is displayed next to the insertion point, as in the example on the right. |
Now type the letter "a." The letter "k" is replaced with the hiragana syllable "ka." If you had typed the letter "i" instead of the letter "a," the hiragana syllable "ki" ({) would have appeared. |
To convert the syllable "ka" into kanji, press the Spacebar. |
Suppose you are looking for a different kanji representation of "ka." If the character you are seeking is not displayed, you can activate a list of alternatives from the IME by pressing the Spacebar a second time. You can scroll down through this list (known as the "candidate window") by pressing the Spacebar a third time, as shown in the example on the right. After you have highlighted the character you want, press Enter to place it in your document. The IME responds by sending the character to the active window. Then the dotted underline representing the IME composition window disappears. |

You can form a number of kanji characters before pressing Enter. The IME engine will attempt to convert your keystrokes into a "determined string" based on Japanese grammar rules. There are four different conversion modes that allow you some control as to where the IME gets its data to convert. (See Table 5-4.)

Table 5-4 Four different IME conversion modes in Windows XP.
IME Conversion Mode | Description
General | This mode is configured for optimum conversion accuracy when writing formal Japanese, such as business correspondence, essays, and manuals.
Bias for names | This mode is handy for times when the user has to enter a lot of personal and place names (for instance, when entering data into databases and spreadsheets, or when writing address labels).
Bias for speech | This mode is configured for optimum conversion accuracy when writing documents containing a large proportion of informal (conversational) Japanese, such as novels, plays, or e-mails to close friends. If you are using default settings, this mode also allows the user to automatically enter emoticons (for example, "smileys"), which is handy when chatting on message boards or when writing e-mail.
No conversion | This mode enters data exactly as is, without performing any conversion.

How the IME System Works

The IME module in Windows 2000 and Windows XP fits into a larger mechanism for passing user input to applications and, like other input methods, the easiest and safest way of handling input is by using standard system controls such as edit fields and rich edit controls. Unless you are writing an IME package or customizing your IME user interface (UI), all of the IME complexities are taken care of for you if you use standard input APIs.

Whether an input language uses an IME or a keyboard to enter a language is something that is entirely transparent to the user. The procedure is the same whether the user is switching IMEs or Western keyboard layouts. Both actions are accomplished by clicking the language indicator on the taskbar or by entering a shortcut-key combination. Furthermore, it does not matter to an application which input method is used because switching IMEs generates the same messages as switching keyboard layouts: WM_INPUTLANGCHANGEREQUEST (if the IME is not part of TSF) and WM_INPUTLANGCHANGE. Applications can activate specific IMEs by calling ActivateKeyboardLayout. The IMM manages communication between IMEs and applications, serving as the go-between. When the user is typing with the IME, each keystroke posts a WM_IME_COMPOSITION message with the GCS_COMPSTR flag to indicate that there is an update to the composition string. The message's WPARAM value returns the first character of the string, and the rest can be retrieved via the ImmGetCompositionString API with the same GCS_COMPSTR flag. Then when the user presses Enter or clicks a character to place it in a document, the IME, by default, posts a WM_IME_COMPOSITION message with the GCS_RESULTSTR flag. (You can retrieve the committed string with the same API and the GCS_RESULTSTR flag.) If the latter WM_IME_COMPOSITION message is sent to DefWindowProc, then for each character in the committed string it posts a WM_IME_CHAR message containing the actual character. For a non-Unicode window, if the WM_IME_CHAR message includes a double-byte character and the application passes this message to DefWindowProc, the IME converts this message into two WM_CHAR messages, each containing one byte of the double-byte character. If the application ignores either message, it falls through to the application's DefWindowProc, which in turn notifies the IMM that the message has been ignored. The IME then resends the character or string byte-by-byte via multiple WM_CHAR messages. For Unicode windows, WM_IME_CHAR and WM_CHAR are identical.

Discussed in the following sections are the three discrete levels of IME support for applications running on Windows: no support, partial support, and fully customized support. Applications can customize IME support in small ways-by repositioning windows, for example-or they can completely change the look of the IME UI.

No IME Support IME-unaware applications basically ignore all IME- specific Windows messages. Most applications that target single-byte languages are IME-unaware.

Applications that are IME-unaware inherit the default UI of the active IME through a predefined global class, appropriately called "IME." This global class has the same characteristics as any other Windows-based common control. For each thread, Windows 2000 and Windows XP automatically create a window based on the IME Global class; all IME-unaware windows of the thread share this default IME window. When IME-unaware applications pass IME-related messages to the DefWindowProc function, DefWindowProc sends them to the default IME window.

Partial IME Support IME-aware applications can create their own IME windows. Applications with partial IME support can use this application IME window to control certain IME behavior. For example, by calling the function ImmIsUIMessage, an application can pass messages related to the IME's UI to the application IME window, where the application can process them. The following code (with proper error handling and possibly more messages handled) would appear in the window procedure of the application's IME window:

HIMC hIMC;
LPVOID lpBufResult;
COMPOSITIONFORM cf;
DWORD dwBufLen;
if (ImmIsUIMessage(hIMEWnd, uMsg, wParam, lParam) == TRUE)
{
switch(uMsg)
    {
    case WM_IME_COMPOSITION:
        if (lParam & GCS_RESULTSTR
        {
           hIMC = ImmGetContext(hWnd);

           dwBufLen = ImmGetCompositionString(hIMC,
                         GCS_RESULTSTR, NULL, NULL) +
                         sizeof(TCHAR);

           lpBufResult =  malloc(dwBufLen);

           if(ImmGetCompositionString(hIMC, GCS_RESULTSTR,
              lpBufResult, dwBufLen) > 0)
           {
              // ...
              // process the text in lpBufResult
              // ...
           }
           else // a negative error value was returned
           {
              // ...
              // handle an error
              // ...
           }
         free(lpBufResult);
         ImmReleaseContext(hWnd, hIMC);
      }
   break;
   }
}
return 0;
}

The same window procedure could call SendMessage either to reposition the status, composition, or candidate windows, or to open or close the status window.

SendMessage(hIMEWnd, WM_IME_CONTROL,
   IMC_SETCOMPOSITIONWINDOW, &cf);

Other API functions that allow the application to change window positions or properties are ImmSetCandidateWindow, ImmSetCompositionFont, ImmSet-CompositionString, ImmSetCompositionWindow, and ImmSetStatusWindowPos. Applications that contain partial support for IMEs can use these functions to set the style and the position of the IME UI windows, but the IME dynamic-link library (DLL) is still responsible for drawing these windows-the general appearance of the IME's UI remains unchanged.

Full IME Support In contrast, applications that are fully IME-aware take over responsibility for painting the IME windows (the status, composition, and candidate windows) from the IME DLL. Such applications can fully customize the appearance of these windows, including determining their screen position and selecting which fonts and font styles are used to display characters in them. This is especially convenient and effective for word processing and similar programs whose primary function is text manipulation and which, therefore, benefit from smooth interaction with IMEs, creating a "natural" interface with the user. The IME DLL still determines which characters are displayed in IME composition and candidate windows, and it handles algorithms for guessing characters and looking them up in the IME dictionary. FULLIME, which is an example of a customized IME UI, can be found in the Microsoft Windows Platform SDK, available at http://msdn.microsoft.com.

Applications that are fully IME-aware trap IME-related messages in the following manner:

  1. They call GetMessage to retrieve intermediate IME messages.
  2. They process these messages in the application WindowProc.
  3. They call TranslateMessage (part of the IMM) to pass the messages to the IME DLL. The IME needs to remain synchronized in the same way that keyboard drivers need to remain synchronized with dead keys. Remember that partial IME support is taken care of for you if you use standard input calls like those to Rich Edit.

You've made sure your application can handle different input languages and methods. Another task in ensuring your application can support multilingual input, output, and display is to meet the inherent demands that complex scripts present. In the sections that follow, you will see various linguistic traits that are associated with complex scripts, and you will learn about Windows support for working with complex scripts.


Complex Scripts

All language versions of Windows 2000 and Windows XP are enabled for all supported languages, thereby empowering applications that use Unicode as their encoding model to handle mixed text from any of the supported scripts. For example, in Notepad you can display text containing English, Farsi, Greek, Hindi, Korean, and Thai text all at once. (See Figure 5-10.) Among these scripts there are several that require special processing to display and edit because the characters are not laid out in a simple linear progression from left to right, as most European characters are. These writing systems are referred to as "complex scripts."

Figure 5-10 A multilingual document that contains different scripts in the same context.
Figure 5-10 - A multilingual document that contains different scripts in the same context.

Characteristics of Complex Scripts

Glossary
  • Item: A character string having all the same script and direction attributes.
  • Runs: Portions of an item that have continuous formatting attributes.
  • Clusters: The sequence of characters or glyphs between points at which the Unicode representation of a string aligns with the glyph representation. For simple text, where each code point is represented by a single glyph, the cluster is the character and its glyph. For simple ligatures, where two or more code points are represented by a single glyph, the cluster is the sequence of code points and the single glyph. The most complex case is when a sequence of code points is represented by a sequence of glyphs with no internal alignment between characters and glyphs. This can occur, for example, in the case of reordering within Indic syllables.

Complex scripts offer significant challenges because of the linguistic traits associated with them. Some of these traits include bidirectionality and character reordering, contextual shaping, characters that combine, and special rules for word breaking, for line breaking, and for text justification.

Bidirectionality and Character Reordering

Arabic scripts (used for languages such as Arabic, Farsi, Pashtu, and Urdu) and Hebrew scripts (used for Hebrew and Yiddish) are not only right-justified, but also are read from right to left (RTL)-in other words, these two scripts have an RTL reading order. For these particular scripts, the logical order (the order in which the user enters text with a sequence of virtual-key inputs) and the visual order (the order in which characters are represented to the user) are different in most cases. (See Figure 5-11.) Character positioning and caret movement in bidirectional context, in which RTL characters and left to right (LTR) characters coexist, are the biggest hurdles to overcome when dealing with RTL scripts. A bidirectional context can be a mixture of Latin and Arabic or Hebrew text; or it can involve Arabic and Hebrew characters with numerals that have an LTR attribute in Arabic and Hebrew. Figure 5-12 shows a challenge that occurs when the trailing edge of one character in bidirectional text is not necessarily adjacent to the leading edge of the next character.

Figure 5-11 Bidirectional text (Arabic) where the logical order (first row) and the visual order (second row) are not of the same sequence of characters.
Figure 5-11 - Bidirectional text (Arabic) where the logical order (first row) and the visual order (second row) are not of the same sequence of characters.
Figure 5-12 In bidirectional text the trailing edge of one character is not necessarily adjacent to the leading edge of the next character. In this example, the text selection follows the logical order.
Figure 5-12 - In bidirectional text the trailing edge of one character is not necessarily adjacent to the leading edge of the next character. In this example, the text selection follows the logical order.

The Unicode bidirectional algorithm resolves the layout of mixed-direction text in the absence of higher-level protocols. The following are some of the general assumptions this algorithm makes:

  • Adjacent runs of words of opposite language direction are laid out according to the base level-left to right for an English paragraph, right to left for an Arabic paragraph.
  • Numbers following LTR words should be displayed to the right of the words.
  • Numbers following RTL words should be displayed to the left of the words.
  • Punctuation between words of the same language direction should be displayed between those words.
  • Punctuation between runs of words of opposite language direction appears between those runs.
  • Punctuation at the beginning or end of a paragraph is laid out according to the paragraph direction and is not affected by the direction of adjacent text.
  • The digits of numbers are laid out left to right in the number.
  • Commas and periods are considered part of a number when immediately surrounded by digits. Other characters, such as currency signs, are considered part of a number when immediately adjacent to a digit.

The algorithm makes a valiant and surprisingly successful stab at resolving what can be very ambiguous text. In applications such as databases and forms, this algorithm is often sufficient. In applications such as word processors, it is usually considered necessary to give the user more direct control over bidirectional-text layout. (To learn more about bidirectionality as well as Unicode algorithms and implementation, go to the Unicode site at http://www.unicode.org.)

Contextual Shaping

For the Arabic and Indic family of languages, a character's glyph (that is, all the character's different possible representations) can change greatly depending on the glyph's position within a word and the characters that precede or follow the glyph. In Arabic, the same character can have several different shapes depending on the context. (See Table 5-5.)

Table 5-5 Arabic letter "Ain" (Unicode code point U+0639) in its four different forms.

Arabic Letter "Ain" | Forms
 | Isolated form
 | Initial form (beginning of a word)
 | Middle form
 | Final form (end of a word)

The difficult part of contextual shaping is that, for all the various glyphs, there is only one defined code point in different encoding models. Layout and displaying mechanisms should define (at run time) the appropriate glyph to be used from the font tables, depending on the context.

Combining Characters

For Latin script, there is often a direct one-to-one mapping between a character and its glyph. (For instance, the character "h" is always represented by the same glyph "h.") For complex scripts, several characters can combine together to create a whole new glyph independent of the original characters. There are also cases where the number of resulting glyphs can be bigger than the original number of characters used to generate those glyphs. Characters are often stacked or combined to create a cluster, which is indivisible for most of the complex scripts. In Arabic, on the other hand, a cluster can be divided by breaking a glyph into its composing characters and diacritics. (See Figure 5-13.)

Figure 5-13 - Cluster formatting for (from left to right) Hindi, where four individual characters get resolved to one indivisible cluster of one glyph; Tamil, where two individual characters get resolved to one indivisible cluster of three glyphs; Arabic, where two individual characters get resolved to one divisible cluster of one glyph.

An indivisible cluster is treated as a single entity in UI-bound text handling. When selected or deleted, an indivisible cluster is selected or deleted as one symbol; when a caret is moved over such a cluster, it skips over the cluster in one cursor move. Divisible clusters, on the other hand, allow the user to position a caret within a cluster and to delete characters that were combined in the cluster.

Word Breaking and Line Breaking

Word breaking and line breaking for Latin script follow some straightforward rules, such as breaking a line at a space, tab, or hyphen. For languages like Thai and Khmer, words run together (with no space between characters that end a word and those that begin another word, as with Latin script). This makes word breaking in such languages a more complex process, since syntax rules require line breaking on word boundaries. Thus for languages like Thai and Khmer, word breaking is based on grammatical analysis and on word matching in dictionaries during text processing at run time.

Text Justification

To justify Latin text, spaces are added between words and characters. This approach cannot be used to justify Arabic text or the contextual shaping will break. Instead, continuous lines (or kashidas) are inserted between adjoining characters to make each word look longer. Figure 5-14 shows an example of Arabic text with kashidas inserted for justification purposes.

Figure 5-14 Arabic text with kashidas (in gray) inserted for justification purposes.
Figure 5-14 - Arabic text with kashidas (in gray) inserted for justification purposes.

In "Techniques for Handling Input Languages" earlier in this chapter, you saw the range of system support available for handling various input languages. Similarly, Windows 2000 and Windows XP offer considerable support to help you work with the intricacies of complex scripts.

Windows Support for Complex Scripts

Glossary
  • Graphics Device Interface (GDI) : In Windows, a graphics display system used by applications to display or print bitmapped text (TrueType fonts), images, and other graphic elements. The GDI is responsible for drawing dialog boxes, buttons, and other elements.

Supported complex scripts in Windows XP are Arabic, Divehi, Hebrew, Syriac, Thai, Vietnamese, and the Indic family of scripts including Bengali, Devanagari, Gujarati, Gurmukhi, Kannada, Oriya, Tamil, and Telugu. The Unicode Script Processor (USP10.dll), also known as "Uniscribe," is the system engine used to shape and lay out complex scripts. It is shipped with Windows 2000 and Windows XP, Microsoft Internet Explorer 4 and later, and with Microsoft Office 2000 and Microsoft Office XP. (For more information on Uniscribe, see "Uniscribe" later in this chapter.)

Although applications can directly interface Uniscribe to render complex scripts, the easiest and most efficient way of supporting complex scripts is through the support that Windows 2000 and Windows XP provide automatically, as discussed in the following section. Each time the system's User or GDI components are called to perform a text output, if support for complex scripts is installed, the text is passed to a language module called "LPK.dll" (a name derived from [L]anguage [P]ac[K]). This module analyzes the text and looks for complex-script portions or runs. If the text does not contain any such runs, it is immediately returned back to the User components or GDI components for processing. If the text does contain complex-script runs, its processing is given to Uniscribe, which sends the final processed text to the User components or GDI components in order to be displayed. (See Figure 5-15.)

Figure 5-15 Flowchart of complex-script processing.
Figure 5-15 - Flowchart of complex-script processing.

As you have discovered, complex scripts often require special processing to display and edit. Luckily, there are additional techniques available to ensure that you overcome the common hurdles you might encounter, as you will see next.


Options for Displaying Text

Now that you have seen some ways to handle complex scripts, the following sections will dig even deeper into how to effectively display text. Covered are ways to display text in Win32 applications and in Web content.

Text Layout in Win32 Applications

In order to display text in a multilingual context, which also entails the output of complex scripts, there are four possible options:

  • Calling Win32 text APIs
  • Instantiating Win32 standard edit controls
  • Instantiating rich edit controls
  • Calling Uniscribe

The following sections briefly explain the advantages of each of these possibilities. It's then up to you to decide which is the best option for your application based on the application's complexity and its design features.

Win32 Text APIs

Many applications deal mostly with plaintext-text that is all in the same typeface, weight, color, and so on. Such applications have traditionally displayed text using standard Win32 display entry points (TextOut, ExtTextOut, TabbedTextOut,and DrawText) to write text to a window, and the GetTextExtent family of functions to measure line lengths. In Windows 2000 and Windows XP, the standard entry points have been extended to support display of multilingual Unicode text and complex scripts, to display vertical text, and to handle special rules regarding line breaking and word breaking. In general, this support is transparent to the application itself, so properly designed applications require no changes to support complex scripts through these interfaces.

Figure 5-16 shows how ExtTextOut can be used to lay out multilingual Unicode text including complex scripts. There is no need for you to do anything other than call ExtTextOut; ithandles everything for you.

Figure 5-16 Multilingual text output using the ExtTextOut API.
Figure 5-16 - Multilingual text output using the ExtTextOut API.

The code looks like the following:

HDC    hDC;
HFONT  hFont;
// Creating a font object to display text using
//    Microsoft Sans Serif.
hFont = CreateFont(14, 0, 0, 0,
   FW_NORMAL, FALSE, FALSE, FALSE, DEFAULT_CHARSET,
   OUT_CHARACTER_PRECIS, CLIP_DEFAULT_PRECIS, PROOF_QUALITY,
   VARIABLE_PITCH | FF_SWISS, TEXT("Microsoft Sans Serif"));
hDC = GetDC(hDlg);
SelectObject(hDC, hFont);

// Outputting buffer lpszText into the selected
//    device context.
ExtTextOut(hDC, 10, 10, ETO_CLIPPED, NULL,
   lpszText, _tcslen(lpszText), NULL);
ReleaseDC(hDlg, hDC);
DeleteObject(hFont);

There are three requirements for displaying complex scripts correctly using the standard Win32-based applications:

  • First, applications should save characters in a buffer and display the whole line of text at once rather than, for example, calling ExtTextOut on each character as it is typed in by the user. When characters are written out one by one, the complex-script shaping modules cannot determine the context for correct reordering and glyph shaping. (See Figure 5-17.)
  • Figure 5-17 In the first row the word is being written one character at a time, which does not provide enough information to Uniscribe to do the layout and shaping. In the second row, the string is passed as a whole to the Win32 API, and the final result is properly laid out.
    Figure 5-17 - In the first row the word is being written one character at a time, which does not provide enough information to Uniscribe to do the layout and shaping. In the second row, the string is passed as a whole to the Win32 API, and the final result is properly laid out.
  • Second, applications should use one of the GetTextExtentXXX functions to determine line length rather than computing line lengths from cached character widths. This is because the width of a glyph used to display a character can vary by context. (See Figure 5-18.)
  • Figure 5-18 The width of the shaped string might be shorter or longer than the sum of individual character widths. In the first row, the two Arabic letters Beh and Alef are longer than the width of the two combined characters.
    Figure 5-18 - The width of the shaped string might be shorter or longer than the sum of individual character widths. In the first row, the two Arabic letters "Beh" and "Alef" are longer than the width of the two combined characters.
  • Third, bidirectional-aware applications should make sure that Arabic and Hebrew scripts are rendered using RTL alignment and reading order. You can use the GetTextAlign and SetTextAlign APIs to retrieve and set, respectively, the alignment of the text in a given device context. As for the reading order, calls to ExtTextOut and DrawText should specify the appropriate RTL reading-order flags, which are, respectively, ETO_RTLREADING and DT_RTLREADING. (See Figure 5-19.)
  • Figure 5-19 Bidirectional text output. On the left side, the reading order for the sentence
    Figure 5-19 - Bidirectional text output. On the left side, the reading order for the sentence "123-52 equals 71." is broken. On the right side, the same sentence is being drawn using ETO_RTLREADING to allow the proper reading order.

Depending on the purpose of your application, you might also find it necessary to support vertical text. Although text that is read horizontally from left to right is becoming more common in East Asian countries-text in Japanese technical and business journals, for example, is often printed horizontally-many books, magazines, and newspapers still print text vertically.

As Figure 5-20 shows, displaying text vertically does not mean that you simply rotate an entire line of text by 90 degrees. Most characters remain upright, but others, such as those identified by arrows, change orientation.

Fortunately, with Win32 you do not need to write code to rotate characters. To display text vertically on Windows 2000 and Windows XP, enumerate the available fonts as usual, and select a font whose font face name begins with the at sign (@). Then create a LOGFONT structure, setting both the escapement and the orientation to 270 degrees. Calls to TextOut are the same as for horizontal text.

Figure 5-20 Text displayed vertically.
Figure 5-20 - Text displayed vertically.

The Windows Platform SDK contains a sample application called "TATE" (short for "tategaki," meaning "vertical writing"), which demonstrates how to create fonts and display vertical text. (For more information on vertical writing, see the Windows Platform SDK documentation, available at http://msdn.microsoft.com.)

Finally, Win32 display entry points can be of enormous help for things such as line breaking and word breaking. Line-breaking and word-wrapping algorithms are important to text parsing as well as to text display. In Western languages, as stated in "Word Breaking and Line Breaking" earlier in this chapter, line breaking occurs at a hyphen, space, tab, or on word boundaries. Word breaking is generally based on white space (spaces, tabs, the end of a line, punctuation, and so on).

Again, the line-breaking and word-breaking rules for Asian languages, however, are quite different from those for Western languages. For example, unlike most Western written languages, Chinese, Japanese, Korean, and Thai do not necessarily indicate the distinction between words by using spaces. Although the Thai language does not use spacing between words, it still requires lines to be broken on word boundaries.

For these languages, world-ready software applications cannot conveniently base line-breaking and word-wrapping algorithms on a space character or on standard hyphenation rules. They must follow different guidelines.

Take Japanese, for example. Japanese line breaking is based on the kinsoku rules-you can break lines between any two characters, with several exceptions. The first exception is that a line of text cannot end with any leading characters-such as opening quotation marks, opening parentheses, and currency signs-that shouldn't be separated from succeeding characters. The second exception is that a line of text cannot begin with any following characters-such as closing quotation marks, closing parentheses, and punctuation marks-that shouldn't be separated from preceding characters. The third exception is that certain overflow characters (such as punctuation characters) are allowed to extend beyond the right margin for horizontal text or below the bottom margin for vertical text.

As you can see, these rules and exceptions can become somewhat complicated. However, by using the appropriate APIs (DrawTextEx, ExTextOut, TextOut, and so on), you don't need to worry about which rule to use. Win32 functions take care of line breaking and word breaking for you. (For more information on East Asian line and word breaking, go to http://www.microsoft.com/globaldev/dis_v1/html/S24B6_L.asp.)

Standard Edit Control

The second option to display text in a multilingual context is to instantiate the standard edit control. This control has been extended in Windows 2000 and Windows XP to support data containing multilingual text and complex scripts, and includes not only input and display, but also correct cursor movement over character clusters (in Thai and Devanagari script, for example). As with the standard Win32 API functions, a well-written application will receive this support automatically, without modification. Again, you should consider adding support for right-to-left reading order and right alignment. In this case, toggle the extended style flags of the edit control window to manage these attributes, as shown in the following code:

// ID_EDITCONTROL is the control ID in the resource file.
HANDLE hWndEdit = GetDlgItem(hDlg, ID_EDITCONTROL);
LONG lAlign = GetWindowLong(hWndEdit, GWL_EXSTYLE);

// To toggle alignment
lAlign ^= WS_EX_RIGHT;

// To toggle reading order
lAlign ^= WS_EX_RTLREADING;

After setting the lAlign value, enable the new display by setting the extended style of the edit control window as follows:

// This assumes your edit control is in a dialog box. If not,
//    you can  get the edit control handle from another source.
SetWindowLong(hWndEdit, GWL_EXSTYLE, lAlign);
InvalidateRect(hWndEdit, NULL, FALSE);

One new feature of the standard edit control is a context menu (activated by pressing the right mouse button while the cursor is in the field) that allows the user to toggle the reading order and to insert or display Unicode bidirectional control characters. (See Figure 5-21.)

Figure 5-21 Edit controls context menu allows the user to insert Unicode control characters and to toggle text reading order.
Figure 5-21 - Edit controls context menu allows the user to insert Unicode control characters and to toggle text reading order.

Rich Edit Control

A third option for multilingual text display is Rich Edit. Rich Edit 3 is a higher-level collection of interfaces that takes advantage of Uniscribe to further insulate text-layout clients from the complexities of certain scripts. Rich Edit provides fast, versatile editing of rich Unicode multilingual text and simple plaintext. It includes extensive message and Component Object Model (COM) interfaces, text editing, formatting, line breaking, simple table layout, vertical text layout, bidirectional-text layout, Indic and Thai support, a Word-like edit UI, and Text Object Model (TOM) interfaces. Rich Edit is the simplest way for a client to support features of complex scripts. Clients use its TextOut function to automatically parse, shape, position, and break lines. (For more information on Rich Edit, see Chapter 21, "Rich Edit.")

Uniscribe

The last of the four options, Uniscribe supports the complex rules found in scripts such as Arabic, Thai, and scripts used for Indic languages. Uniscribe also handles scripts written from right to left, such as Arabic or Hebrew, and supports the mixing of scripts. For plaintext clients, Uniscribe provides a range of ScriptString functions that are similar to TextOut, with additional support for caret placement. The remainder of the Uniscribe interfaces provide finer control to clients.

As stated in "Windows Support for Complex Scripts" earlier in this chapter, Uniscribe uses multiple shaping engines that contain the layout knowledge for particular scripts. It also takes advantage of the OpenType layout shaping engine for handling font-specific script features such as glyph generation, extent measurement, and word-breaking support.

Uniscribe subdivides strings of characters into items, runs, and clusters. The client builds runs based on its own stored formatting attributes and on the item boundaries obtained by calling the Uniscribe ScriptItemize API. The Uniscribe ScriptShape API breaks a run into clusters according to script rules and then generates glyphs. The ScriptPlace API generates x and y positions for the characters. The ScriptTextOut API then displays the glyphs using the x and y positions.

Uniscribe supports line breaking at word boundaries through ScriptBreak. Hit testing and cursor positioning are supported by ScriptCPtoX and ScriptXtoCP. Character-to-glyph mapping is provided by ScriptGetCMap. Uniscribe manages bidirectional character reordering using the Unicode bidirectional algorithm, and also understands non-OpenType layout font formats for Arabic, Hebrew, and Thai shaping and positioning.

Using Uniscribe, the text-layout client only needs to manage a backing store of Unicode character codes. The client does not need to maintain any other buffer or mapping table to track character order, but rather only needs to store and manage the order in which the characters were entered by the user. This is the same logical order as defined by Unicode. The client's backing store never changes as a result of layout operations. Uniscribe maintains an index from the reordered clusters to the original character boundaries passed by the client.

Uniscribe is a single API for Unicode output across Microsoft's operating-system range. (For more information on coding techniques with Uniscribe, see Chapter 24, "Uniscribe," and Cssamp.exe in the Samples subdirectory on the companion CD. Other resources include the MSDN documentation and the article "Multilanguage text support in Windows 2000" at http://microsoft.com/globaldev/articles/multilang.asp.)

Again, remember that you can take advantage of Uniscribe's features through the system's standard support in edit controls without having to interface with Uniscribe directly. The only exception is if you need to use advanced text formatting.

As you can see, this chapter provides more in-depth information about displaying text within Win32 applications than it does about text in Web content and in the .NET Framework. This is because the rendering engine of Internet Explorer hides all implementation details in Web content (both traditional Web files and within the .NET Framework), thereby interfacing with Uniscribe in a transparent manner.

Text Input, Output, and Display in Web Content and in the .NET Framework

Text input, output, and display in Web content has been made a lot easier because HTML rendering in Internet Explorer is handled by the Trident module (Mshtml.dll), which is one of the Uniscribe clients. All support for different input languages and complex scripts is provided to Web-based pages automatically and transparently, as long as Unicode encoding (either UTF-8 or UTF-16) is used. For Web content within the .NET Framework, system support hides all implementation details for Microsoft Windows Forms and for other .NET applications.

Another essential aspect of a globalized application is its ability to display the correct font. Thanks to the evolution of font technology, enabling support for varying fonts has become a more manageable task.


Fonts

Although starting out as fixed-width raster fonts that were a collection of bitmaps limited to a given script, fonts evolved later into Unicode-based OpenType fonts containing extensive tables on glyph positioning and substitution. This advancement in font technology has been one of the most important contributions to multilingual computing. In fact, fonts are the final form in which multilingual data is displayed to the user.

In the sections that follow, you will first be introduced to some of the new technologies used in Windows 2000 and Windows XP in order to accommodate font support and multilingual text display. Then you will learn about different programming techniques that you should use to take advantage of the system's support for fonts both in Win32 and Web content.

Font Support in Windows 2000 and Windows XP

Glossary
  • TrueType: A digital font technology (designed by Apple Computer and now used by both Apple and Microsoft in their operating systems) that offers superior display quality on computer screens and printers.
  • Positional forms: Refers to the shape of a character that varies with the character's position in a word. For example, the Arabic character "ha" can take any of four shapes, depending on whether the character stands alone or whether it falls at the beginning, middle, or end of a word.
  • Alternates: Similar to positional forms. For example, kanji uses alternative forms of parentheses when positioned vertically.
  • Substitutions: English, French, and other languages based on Latin can substitute a single ligature, such as "fi," for this particular ligature's component glyphs "f" and "i." Conversely, the individual "f" and "i" glyphs could replace the ligature, possibly to give a text-processing application more flexibility when spacing glyphs to fill a line of justified text.
  • Vertical metrics: A collective term to indicate information from a set of fields that is available in various font tables for determining vertical spacing when laying out text. Some of these fields can be used to arrive at the vertical distance required between two consecutive horizontal baselines, while others can be used to determine vertical spacing for each glyph when text is laid out vertically.

One of the biggest challenges in enabling the operating system for international character sets is the ability to select and display the right glyph. When editing a multilingual document, the user should not be expected to select a different font for each one of the scripts he or she wants to view because:

  • The average user might not know which font is the most suitable choice.
  • Simple applications such as Notepad.exe only allow one font for the whole document.
  • This type of font selection would impose a big productivity overhead.

Therefore, in addition to the font substitution (also known as "font association") technique used since the early versions of Windows, new features-OpenType fonts, font fallback, and font linking-were introduced in Windows 2000 to solve these types of font-selection problems.

OpenType Fonts

The Unicode-based OpenType font format has been developed jointly by Microsoft and Adobe; it extends the TrueType font file format originally designed by Apple. OpenType fonts allow rich mapping between characters and glyphs, thus enabling support for ligatures, positional forms, alternates, and other substitutions. OpenType fonts can also include information that supports two-dimensional glyph positioning and glyph attachment, and can contain either TrueType or PostScript outlines. Layout features within OpenType fonts are organized by scripts and languages, allowing a single font to support multiple writing systems, even within the same script.

Core OpenType fonts (such as Tahoma or Arial) contain glyphs for Western and Central European, Hebrew, Arabic, Greek, Turkish, Baltic, Cyrillic, and Vietnamese scripts. Although OpenType fonts do not contain East Asian script characters, they link to fonts that do. The main reason behind the exclusion of these scripts is related to the massive performance overhead that East Asian glyphs would introduce in terms of font loading and mapping in GDI. In addition, these scripts would make the font size several times bigger. Instead of having instructions on how to create glyphs for several hundred characters, you would have instructions on how to create them for some 6,000 or 7,000 characters, approximately. (For more information on OpenType font technology, see Chapter 20, "OpenType Fonts.")

Font Fallback

One benefit of Unicode is the ability to represent many languages and scripts in a single string. This is also a problem, since very few fonts support more than a couple of scripts. Indeed, it's very difficult to do a good job of making fonts with glyphs for different scripts such that all conform to one set of vertical metrics. To overcome this limitation, and in order to accommodate complex scripts, Uniscribe can detect if the currently selected font doesn't support a particular script and can automatically switch-or fall back-to a predefined system font that has appropriate glyphs for the desired script. All these operations are transparent to the user.

Here is an example to better understand this mechanism. A user running Windows XP selects the Tahoma font to enter some text first in English, next in Hebrew, and then in Telugu. Since Tahoma is an OpenType font, it provides support for Latin and Hebrew scripts, but does not contain any Telugu glyphs. Uniscribe detects this lack of font support and automatically renders the Telugu script by using its fallback font, which is Gautami.

Although font fallback can accommodate Indic scripts such as Telugu in Windows XP, no such mechanism existed in Windows 2000. For most of the scripts, the fallback font is set to Microsoft Sans Serif (an OpenType font). For the Indic family of languages, the fallback is set to another appropriate font. Font fallback is internal to Uniscribe, and applications cannot add new fallback fonts or modify existing ones.

Font Linking

Unlike font fallback, in which the selected font is internally replaced by a predefined font, in font linking it is possible to link one or more fonts (called "linked fonts") to another font (called the "base font"). Once you link fonts, you can use the base font to display code points that do not exist in the base font, but that do exist in one of the linked fonts. For example, linking a hangul font and a Japanese font to a Tahoma font allows you to display both Korean and Japanese characters in Tahoma font.

Note
Font linking can only add glyphs to a base font; you cannot override or replace glyphs in the base font.

If font linking is enabled on your device, you can examine the registry by enumerating the subkeys of the registry key at HKEY_LOCAL_MACHINE-\SOFTWARE\Microsoft\Windows NT\CurrentVersion\FontLink\SystemLink to determine the mappings of linked fonts to base fonts. You can add links by using Regedit to create additional subkeys. Once you have located the registry key that has just been mentioned, from the Edit menu, click New and then click String Value. Replace <New Value #1> with the <base font face name.> Highlight the new name and then from the Edit menu, click Modify. Within the Edit String dialog box, for the dialog field Value data, enter <face name of the font to link to>.

Important
Font linking is a mechanism enabled within GDI and takes priority over font fallback.

With font fallback and font linking, the font size of the newly selected font will be the same as that of the original font. For example, if an 8-point Tahoma font was selected to type English and now the user enters some Japanese text, an 8-point MS UI Gothic font will be automatically selected. The 8-point font size might not be the best choice for some scripts, since it can make them hard to read.

Both font fallback and font linking contain logic to estimate an appropriate font size, but both mechanisms have to use metrics exposed by the font that might or might not actually match the way the font appears. Consider the difference in the height of English letters among 8-point Microsoft Sans Serif, 8-point Traditional Arabic, and 8-point Angsana New ( [image] respectively). Even though all of these are supposedly 8-point fonts, the actual size of the English letters varies widely. Font fallback and font linking are no substitutes for choosing the right font in the first place. Rather, these mechanisms are simply a means of preventing the user from manually selecting a font; additionally, they prevent UI text from being displayed as a default glyph.

The following sections contain technical recommendations for font selection in Win32 applications. More specifically, you'll see best practices for dealing with dialog resource files and for selecting fonts at run time.

Font Selection in Win32 Applications

Fonts that have glyphs for all supported scripts and for all Unicode characters are very rare. Arial Unicode MS is one of the most complete fonts for Windows, and yet it does not contain all the glyphs associated with all Unicode code points. In order for your application to take advantage of the system font support described previously, you should, as a general rule, adhere to the following guidelines:

  • Do not hard-code font face names. Font face names might have variations to specify the default character set (charset). For example, the Arial font face name on an Arabic machine is known as "Arial, 178."
  • Do not assume a given font is installed. The user might delete or uninstall fonts (even a system font!).
  • Do not assume a selected font supports the desired script. For example, it is impossible to use Miriam (a Hebrew font) to represent hiragana script.

Along the same lines, it's strongly suggested that you do not hard-code the font size that you use, and that you make this variable customizable according to the script to be displayed; since some scripts are more complicated than others, they need more pixels to be displayed properly. For example, most English characters can be displayed on a 5x7 grid, but Japanese characters need a grid of at least 16x16 to be seen clearly. Chinese characters, on the other hand, need a 24x24 grid. Thai characters only need 8 pixels for width, but they need at least 22 pixels for height. Thus it is easy to understand why some characters in a small font size might not be legible. (See Figure 5-22.)

Figure 5-22 Comparison of English and Japanese characters in varying font sizes.
Figure 5-22 - Comparison of English and Japanese characters in varying font sizes.

Fonts in Dialog Resource Files

System fonts are different from one language version to another (even within the same version of the operating system). Those in charge of translating resource files often do not have enough information or technical background on how to change the font face name-whether this change involves replacing the entire name or only modifying it slightly, such as when adding charset information-for the different languages into which they are translating text. In the following example, MS Sans Serif, a bitmap font that only contains glyphs for Western European languages, is being used in the dialog resources. If the application is localized into Turkish or Japanese, for example, without changing the font face name, the UI text will be displayed in the default glyph as empty squares. (See Figure 5-23.) This type of display occurs because MS Sans Serif is not an OpenType font that can accommodate Turkish script, and it is not font linked in the system.

DLG_NLS DIALOG DISCARDABLE 0, 0, 344, 260
STYLE DS_MODALFRAME | WS_POPUP | WS_CAPTION | WS_SYSMENU
CAPTION "NLS APIs"
FONT 8, "MS Sans Serif"
Figure 5-23 A property sheet translated into Korean on Windows 2000, where the font is set to MS Sans Serif. The system fails to find appropriate glyphs for this font and ends up displaying the default glyph.
Figure 5-23 - A property sheet translated into Korean on Windows 2000, where the font is set to MS Sans Serif. The system fails to find appropriate glyphs for this font and ends up displaying the default glyph.

Because the desired behavior is to have the UI font of your application follow the desktop (Shell) UI font, and because the default Shell font is different from one localized language of the operating system to another (for example, Microsoft Sans Serif for English, Tahoma for Arabic, and so on), the best practice is to always use the higher-level font face name known as "MS Shell Dlg." MS Shell Dlg is actually not a font. Rather, it is a font face name that gets mapped to the right font depending on the font-substitution settings of the operating system. By setting your default resource font as MS Shell Dlg, you are assured of providing the appropriate font solution, not only on Windows 2000 and Windows XP, but also on all versions of Windows since Windows 95!

Font Selection at Run Time

Hard-coding the font name in your code can have the same result as hard-coding it in dialog resource files. Either action can break your UI. Here again, your font selection should be flexible and context-dependent. Instead of a direct call to the CreateFont or CreateFontIndirect APIs, where the font attributes are hard-coded in a LOGFONT structure, you should use EnumFontFamiliesEx. EnumFontFamiliesEx enumerates all fonts in the system that match the font characteristics specified by the LOGFONT structure-in this case, by character set instead of by font face name.

Possible character sets for a LOGFONT structure are shown in Table 5-6.

Table 5-6 Possible character sets for a LOGFONT structure.

Charset Name | Charset Value | Charset Value | Code Page
 | (Hex) | (Decimal) | ID
ANSI_CHARSET | 0x00 | 0 | 1252
DEFAULT_CHARSET | 0x01 | 1 | CP_ACP
SYMBOL_CHARSET | 0x02 | 2 | CP_OEMCP
SHIFTJIS_CHARSET | 0x80 | 128 | 932
HANGUL_CHARSET | 0x81 | 129 | 949
GB2312_CHARSET | 0x86 | 134 | 936
CHINESEBIG5_CHARSET | 0x88 | 136 | 950
GREEK_CHARSET | 0xA1 | 161 | 1253
TURKISH_CHARSET | 0xA2 | 162 | 1254
HEBREW_CHARSET | 0xB1 | 177 | 1255
ARABIC_CHARSET | 0xB2 | 178 | 1256
BALTIC_CHARSET | 0xBA | 186 | 1257
RUSSIAN_CHARSET | 0xCC | 204 | 1251
THAI_CHARSET | 0xDE | 222 | 874
EE_CHARSET | 0xEE | 238 | 1250
OEM_CHARSET | 0xFF | 255 | n/a

Notice that DEFAULT_CHARSET is not a real charset; in reality on Windows 2000 and Windows XP it does two things:

  • It tries to select the named font with the current system character set.
  • If the named font exists but does not support the system character set, it will still select the font with a charset that the font does support.

DEFAULT_CHARSET should be used when displaying a string of characters encoded with Unicode. In the example that follows, the code identifies the charset corresponding to the currently selected input language and also enumerates a set of compatible fonts. hWnd is the window handle where the font will be used, and hDlg is a dialog box to display the font list.

DWORD   dwCodePage;
HKL     hkl = (HKL) lParam;
LOGFONT lf;
HDC     hDc;
CHARSETINFO cs;
TCHAR szLocaleData [BUFFER_SIZE];
// Initialize the LOGFONT to be used.
_tcscpy (lf.lfFaceName, TEXT(""));
lf.lfCharSet = DEFAULT_CHARSET;

// This is a workaround for Hindi and Tamil, since they
//    don't have charsets. Mangal and Latha are the
//    fonts for Hindi and Tamil shipping with Windows NT,
//    2000, and XP.  A better workaround would be to put
//    these strings in data files that can be updated with new
//    font face names. You would then call
//    EnumFontFamiliesEx once per face name.
if (LOWORD(hkl) == MAKELANGID(LANG_HINDI, SUBLANG_DEFAULT))
     _tcscpy (lf.lfFaceName, TEXT("Mangal"));
else if (LOWORD(hkl) == MAKELANGID(LANG_TAMIL, SUBLANG_DEFAULT))
        _tcscpy (lf.lfFaceName, TEXT("Latha"));
else
{
   // Find out what Charset the new kbd wants.
   GetLocaleInfo (LOWORD(hkl), LOCALE_IDEFAULTANSICODEPAGE,
      szLocaleData, 6);
   dwCodePage = _ttol (szLocaleData);
   if (TranslateCharsetInfo ((LPVOID) dwCodePage, &cs,
      TCI_SRCCODEPAGE))
      {
         lf.lfCharSet   = (BYTE) cs.ciCharset;
      }
}
// Get list of fonts that support this charset.
// hDc is needed by EnumFontFamilies.
hDc = GetDC (hWnd);

// Callback uses hDlg.
EnumFontFamiliesEx (hDc, &lf, (FONTENUMPROC)EnumFontProc,
   (LPARAM)hDlg, (DWORD) 0);

ReleaseDC (hWnd, hDc);

In this example, the callback function passed to EnumFontFamiliesEx is as follows:

int CALLBACK EnumFontProc (ENUMLOGFONTEX* lpelfe,
      NEWTEXTMETRICEX* lpntme, int iFontType, LPARAMlParam)
{
   // Size computed from format used below and buffer limits
   TCHAR SzFaceName [4+LF_FULLFACESIZE+LF_FACESIZE];
      _stprintf(szFaceName, TEXT("%s (%s)"), lpelfe->elfFullName,
      lpelfe->elfScript);

   // Add string to list box to describe this font.
   SendDlgItemMessage ((HWND)lParam,
   IDC_FONTLIST, LB_ADDSTRING, (WPARAM) 0, (LPARAM) szFaceName);
   return TRUE;
}

Each time the callback function is requested, it builds a string containing the font face name and the language name, and adds that string to the list box (the control name that is part of a dialog template).

As you can see from the previous code sample, Indic scripts must be handled separately because they have no charset values. Since there is no default ANSI or Windows Code Page (ACP) value for Indic scripts, none of the Win32 ANSI entry points (the "A" routines) will work with Indic, Georgian, or Armenian text, or for any new scripts for which system support is provided exclusively through Unicode encoding. (See Chapter 3, "Unicode." )

The font resource for many East Asian languages has two names: an English name and a localized name. For Windows 95, Windows 98, and Windows NT 4, the localized name only works on a system locale that matches the language, while the English name works on all other system locales. This can be a problem when calling CreateFont or CreateFontIndirect. The best method is to try one name and, if that fails, try the other. EnumFonts, EnumFontFamilies, and EnumFont-FamiliesEx returns the English font face name if the system locale does not match the language of the font. On Windows 2000 and Windows XP, this is no longer a problem because both names will be valid for any locale.

Note
In Windows 2000 and Windows XP, you can use both English and localized font face names in CreateFontXXX. However, when you enumerate these fonts using EnumFontXXX, you will only get English font face names if the system locale does not match the font's intended face name language.

Another approach to run-time font selection is to display a font selection common dialog box, from which the user can select the desired font. (See Figure 5-24.) With the ChooseFont API, you can control the list of fonts that are returned to the user, and you can limit the fonts to a given character set.

Figure 5-24 A simplified font selection dialog box.
Figure 5-24 - A simplified font selection dialog box.

The following code example initializes a font for the IDC_EDITWIN edit control. Then upon the user's selection, the code adjusts the font used in the edit control:

static      CHOOSEFONT   cf;
static      LOGFONT      lf;
// Fill out our CHOOSEFONT and LOGFONT and CHOOSEFONT structures
//    with default and predefined values.
InitializeFont(hDlg, &cf, &lf);
// Create this font.
hEditFont = CreateFontIndirect(&lf);

// Set the font in our edit control.
SendDlgItemMessage(hDlg, IDC_EDITWIN, WM_SETFONT,
   (WPARAM) hEditFont, MAKELPARAM(TRUE, 0));
// Upon user's request, create a font selection common
// dialog box and use the new font.
if (ChooseFont(&cf))
{
   hEditFont = CreateFontIndirect(&lf);
   SendDlgItemMessage (hDlg, IDC_EDITWIN, WM_SETFONT,
      WPARAM) hEditFont,  MAKELPARAM(TRUE, 0));
}

Where the InitializeFont function looks like the following:

void InitializeFont(HWND hWnd,  LONG lHeight, LPCHOOSEFONT lpCf,
         LPLOGFONT lpLf)

{
   lpCf->lStructSize   = sizeof(CHOOSEFONT);
   lpCf->hwndOwner     = hWnd;
   lpCf->hDC           = NULL;
   lpCf->lpLogFont     = lpLf;
   lpCf->iPointSize    = 10;
   lpCf->Flags = CF_SCREENFONTS|CF_INITTOLOGFONTSTRUCT|CF_NOSIZESEL;
   lpCf->rgbColors     = RGB(0,0,0);
   lpCf->lCustData     = 0;
   lpCf->lpfnHook      = NULL;
   lpCf->lpTemplateName= NULL;
   lpCf->hInstance     = g_hInst;
   lpCf->lpszStyle     = NULL;
   lpCf->nFontType     = SIMULATED_FONTTYPE;
   lpCf->nSizeMin      = 0;
   lpCf->nSizeMax      = 0;

   lpLf->lfHeight      = 24;
   lpLf->lfWidth       = 0;
   lpLf->lfEscapement  = 0;
   lpLf->lfOrientation = 0;
   lpLf->lfWeight      = FW_DONTCARE;
   lpLf->lfItalic      = FALSE;
   lpLf->lfUnderline   = FALSE;
   lpLf->lfStrikeOut   = FALSE;
   lpLf->lfCharSet     = DEFAULT_CHARSET;
   lpLf->lfOutPrecision= OUT_DEFAULT_PRECIS;
   lpLf->lfClipPrecision = CLIP_DEFAULT_PRECIS;
   lpLf->lfQuality     = DEFAULT_QUALITY;
   lpLf->lfPitchAndFamily = DEFAULT_PITCH | FF_DONTCARE;
   _tcscpy(lpLf->lfFaceName, TEXT("MS Shell Dlg"));
}

By adhering to the guidelines of the following section, you can have better control over font customization in Web pages. You'll also save valuable time.

Font Manipulation in Web Content

When creating Web pages, avoid placing font attribute values into inline styles, as shown below:

<span style = "font-size: 10pt; font-family: Arial;"> Hello </span>

This approach makes font customization per language or script a difficult task, since a technical localizer would need to scan the entire Web content for all instances of the font definition one language at a time. If the font didn't have glyphs to handle the new language, changes would have to be made on a per-language basis.

A better way to handle font attributes is to use cascading style sheets (CSS) in which corresponding font attributes and styles are defined. In the following example, the CSS file creates a style class called "myStyle," which contains the font family and font size. You can allow these attributes to change depending on the language into which you are rendering your content. For the HTML file, all you need to do for the Web page is "span" whatever text you want formatted with the myStyle class.

      <style>
      .myStyle {font-size: 10pt; font-family: Arial;}
      </style>
      <span class = myStyle> Hello </span>

Now adopting the font to be used per language or per script becomes a much easier job, since it requires a single change in one specific file. You can extend this notion and define a specific style for all the scripts that you want to render in your multilingual Web site. In the following example, the CSS file defines an appropriate font style for each script that will be used thereafter in the inline text.

The CSS file would look like this:

.clsDescriptor{COLOR: #bdbddd;FONT: 0.7em/1em Verdana;}
.clsEnglish {FONT: 1.1em/1.3em "Palatino Linotype";}
.clsTitle {COLOR: darkred; FONT: 1.4em/1.6em "Palatino Linotype";}
.clsArabic {FONT: 1.1em/1.3em "Arabic Transparent";}
.clsArmenian {FONT: 1.3em/1.3em Sylfaen;}
.clsHindi {FONT: 1.1em/1.3em Mangal;}

And the HTML file would look like this:

<HTML>
<HEAD>
<TITLE>LANGUAGE SUPPORT IN WINDOWS 2000</TITLE>
<META content="text/html;charset=UTF-8"http-equiv=Content-Type>
<LINK href="css.css"rel=stylesheet type=text/css>
</HEAD>

<BODY bgColor=#ffffff lang=EN-US leftMargin=5 topMargin=30>

<SPAN class="clsTitle">
       Each version of Windows 2000 and Windows XP</SPAN><p>
<SPAN class="clsEnglish">supports hundreds of languages.</SPAN>
<SPAN CLASS="clsDescriptor">[English ]</SPAN><p>
<SPAN class="clsArabic"DIR=rtl>
  </SPAN>
<SPAN CLASS="clsDescriptor">[Arabic ]</SPAN><p>
<SPAN class="clsArmenian">
   </SPAN>
<SPAN CLASS="clsDescriptor">[Armenian ]</SPAN><p>
<SPAN class="clsHindi">  </SPAN>
<SPAN CLASS="clsDescriptor">[Hindi ]</SPAN><p>

</BODY>
</HTML>
Figure 5-25 Output of code in which an appropriate font style has been defined for each script.
Figure 5-25 - Output of code in which an appropriate font style has been defined for each script.


Summary

Making sure that your application can properly handle text input, output, and display is a challenging job if you don't use Windows system-supported APIs. You'll need to enable your application to deal with different input languages and methods, including different keyboard layouts and IMEs, as well as new input services such as voice-recognition engines or handwriting-recognition engines. You will also need to make sure your application properly displays all supported scripts, in accordance with the linguistic characteristics associated with them. These scripts include complex scripts that require special handling in terms of layout and character shaping. Additionally, you will need to understand the different options available for displaying multilingual text-including such things as vertical text-by leveraging system support. Finally, you should be able to handle fonts in a multilingual context.

Considerable effort has been made to provide various system services for applications. By creating your own solution, you risk having an application that is incompatible with the rest of the operating system. Some of the support that Windows 2000 and Windows XP offer include the IME module, whereby a user can enter ideographic characters or phonetic syllables with various methods of input. Windows 2000 and Windows XP also offer support for complex scripts, enabling you to deal with linguistic traits associated with these scripts, such as bidirectionality, character reordering, contextual shaping, combining characters, and special rules in terms of word breaking, line breaking, and text justification.

To effectively display text in a multilingual context for Win32 applications, you can call Win32 text APIs, instantiate Win32 edit controls, instantiate rich edit controls, or call Uniscribe. Through these four options, Windows 2000 and Windows XP provide support for displaying complex scripts and multilingual Unicode text.

To display the correct font, there are, once again, numerous types of system support available. This support for fonts, which includes OpenType fonts, font fallback, and font linking, can help you as you work with fonts in Win32 applications and Web content. For Win32 applications, do not hard-code the font size you use, and do not hard-code the font name in your code, since doing so could break your UI. To effectively display fonts in Web content, avoid placing font attribute values into inline styles. Instead, use CSS in which font attributes and style are defined in private styles. By following the best practices and leveraging the system support discussed in this chapter, you can surmount many of the obstacles that multilingual computing often presents.


Chapter 6: Multilingual User Interface (MUI)

The Indian television network known as "Doordarshan" broadcasts movies all over the country. Since there are 17 national languages and about 180 official languages in India, it is too expensive to produce and broadcast 17 or more subtitled prints of the same movie. Instead, the network uses a single feed, and each regional station gets to choose which subtitles it wants to show. For example, if the network is showing a Hindi movie, viewers in Hyderabad will get subtitles in Telegu, viewers in Karnataka will get subtitles in Kannada, and those in Goa will see subtitles in Konkani. However, viewers in Bihar and Uttar Pradesh will get no subtitles, since Hindi is the official language there.

Similarly, software must also be adaptable to the needs of its international, multilingual market in a way that is efficient and resourceful. The ultimate goal of fully globalized software is to empower the user to access and manipulate data in any language; the mantra "any language, any culture" has been the driving force behind the creation of Microsoft Windows 2000 and Microsoft Windows XP international feature set. Adopting a unified encoding mechanism for all scripts via Unicode was the very first step toward that goal. (See Chapter 3, "Unicode." ) Making sure that your applications are locale-aware and culture-aware-by leveraging Windows globalization services to format locale-sensitive or cultural-sensitive data according to user preference-was the second step. (See Chapter 4, "Locale and Cultural Awareness." ) Handling different input languages, properly designing your application's text output, and dealing with fonts in a multilingual context was the obvious next step. (See Chapter 5, "Text Input, Output, and Display.")

The present chapter is dedicated to the last step of software globalization: preparing the application to determine the language of the user interface (UI)-either through detection of the operating system's UI language or through user selection-and displaying data in the user's preferred language. The chapter first looks at MUI support in Windows 2000 and Windows XP. It then discusses options for offering your own MUI solution for Microsoft Win32 applications, Web content, the Microsoft .NET Framework, and for console applications-with pros and cons of each option. Finally, the chapter provides recommended practices and code samples.


MUI Support in Windows

Before Windows 2000, a user who needed to access the same operating system in different languages was forced to either have several machines on his or her desk-each with a different language version-or to have one machine with some complicated process of allowing the user to start different language versions of an operating system on this same machine. In addition, global corporations needed to support several different language versions of an operating system across their regional offices-much akin to the Indian television network having to broadcast 17 or more subtitled prints of the same movie. At the request of their customers to find a more cost-effective way of handling these two situations, Microsoft added MUI functionality to a particular version of Windows 2000 that was enhanced in the release of Windows XP1 . This functionality allows the UI language of the operating system to be changed, according to user preference, to one of many supported languages.

Availability

This MUI support was released as the Windows 2000 MultiLanguage Version and as the Windows XP Multilingual User Interface Pack. Although it is not available in consumer versions of Windows-Microsoft Windows 95, Microsoft Windows 98, Microsoft Windows Millennium Edition (Me), and Microsoft Windows XP Home Edition-it is available for the Microsoft Windows 2000 Professional and the Windows 2000 Server Family, and is often implemented in a Terminal Services environment. It is also available for English versions of Windows XP Professional, Windows XP Embedded, Microsoft Windows .NET Server Family, and the Microsoft Windows XP Tablet PC Edition. Most of the MUI languages supported correspond to official localized versions2 of Windows 2000 and Windows XP.

The remainder of the supported languages exists only on Windows XP via an extension of the MUI support called "Language Interface Pack" (LIP). This pack contains a partial localization of the more commonly used portions of the UI, which in turn allows a majority of routine operating-system tasks to be localized. (For a full list of the languages supported, see Appendix V, "Localized Versions of Microsoft Windows.")

Advantages of MUI

MUI allows large corporations to roll out, support, and maintain the same worldwide image with a single installation. Other benefits of MUI are that users of different languages can share the same workstation, and roaming users can take their localized UI from one workstation to another. For instance, one user might choose to see system menus, dialog boxes, and other text in Japanese, while another user logging onto the same system might prefer to see the corresponding text in Danish. (There are two ways to set a UI language. Either the user can select it [see Figure 6-1], or an administrator can set it through Group Policy for organizational units. Changing the UI language requires the user to log off and log back on to the operating system in order for changes to take effect.)

Figure 6-1 The Language tab within Regional And Language Options property sheet, with a drop-down menu only available when MUI is installed.
Figure 6-1 - The Language tab within Regional And Language Options property sheet, with a drop-down menu only available when MUI is installed.

A Windows system with MUI set to a particular UI language will largely look and behave like the localized version, with some exceptions. Since MUI runs on top of the English version of Windows, from a feature and architectural point of view, localized versions of Windows are the same as the MUI version on English Windows XP. However, on localized versions:

  • The UI resources are fully localized.
  • Windows Setup information-such as the system locale, user locale, and keyboard layout-is customized for the specific language or country. With the MUI Pack, however, this is a policy setting. By default, installing MUI does not change locale settings of the English system where it is installed.
  • Additional country-specific printer drivers are added in the East Asian versions only.
  • There is support for upgrades from localized versions of Windows 95, Windows 98, Windows Me, or Windows 2000 to localized versions of Windows XP. MUI can be installed only on English versions of Windows; as long as cross-language upgrades are not supported, you cannot upgrade a localized operating system to an English version of Windows with MUI.

There is no difference between the actual translations on a particular localized version of Windows XP, for example, and those on an MUI version. This is because the same resources are used to create both the localized version and the MUI version. As a result, the MUI version is almost fully localized, apart from INF files, UI strings that are stored in the registry, components such as HyperTerminal that are not Unicode-based, and 16-bit applications in code page-based format. In fact, the percentage of localization coverage with MUI in Windows 2000 varies between 90 and 95 percent depending on the language.

In Windows 2000 MultiLanguage Version, there are many UI strings that appear in English, which are particularly noticeable with the Start menu. This is because the Start menu is populated directly by using the file names of folders and link files created at setup time. These names appear in English even if you are running an MUI system with a Japanese UI, since English was the original installation language.

In Windows XP, however, the practice of using English UI strings has been changed, resulting in a much greater degree of localization for the system user (with a localization coverage of about 97 percent). Much of the additional localization coverage in Windows XP is achieved through "MUI-enabling" Windows XP system modules and applications. MUI-enabling essentially entails:

  • Transferring UI strings from the registry to Windows resource files.
  • Removing localizable strings from the kernel.
  • Using the MUI-enabled shell to display localized strings for Start menu items, desktop shortcuts, shell menu items, file-type names and shell verbs (shell's right-click menu items).
  • Making sure that Windows Services that display the UI impersonate the current interactive user rather than using the "system" user, so that UI language preferences can be respected.
  • Preventing the use of hard-coded file paths when loading resource files, including Help files, so that an alternate resource path can be used to load resource files.
  • Providing special code in each component to install and load the UI resource if a nontraditional Win32 resource is used-such as a resource based on Extensible Markup Language (XML) or on HTML.

Implementing these changes for the Windows XP MUI Pack has resulted in much higher localization quality and in a vastly improved user experience, when compared with MUI in Windows 2000. Note, however, that a system running MUI will require more disk space to store the necessary resource files. The needed disk space for the MUI Pack on Windows XP is about 120 megabytes (MB) per language (exact size varies from one language to another). Note that for East Asian languages, Windows features such as fonts and Input Method Editors (IMEs) will also be added if they are not already installed, which could require an extra 150-200 MB of space for these languages. (For more information and details on the MUI Pack, go to http://www.microsoft.com/globaldev/FAQs/Multilang.asp.)

The advantages of using MUI within Windows 2000 and Windows XP in today's multilingual computing environment are obvious. You can apply these same benefits to your own application through several methods.


Options for Offering an MUI Solution

The number of possible ways to implement an MUI solution in applications is endless, but each possibility is a variation of one of three basic methods:

  • Compiling one binary (with built-in resources) for each language
  • Having one language-neutral core binary with one multilanguage resource dynamic-link library (DLL)
  • Having one language-neutral core binary with one set of resource DLLs per language

This section discusses each one of these techniques, along with their pros and their cons. You can offer an MUI solution for your application regardless of the version of Windows. In fact, these techniques allow you to offer your MUI solution even on Windows 95. The following comparison assumes that your application is only targeting three pilot languages: English, French, and Japanese.

Method 1: Language-Dependent Binary

This traditional approach involves compiling one binary, containing both source code and resources, for each target language. Figure 6-2 illustrates the file distribution of such an approach.

Figure 6-2 File distribution for the language-dependent binary approach.
Figure 6-2 - File distribution for the language-dependent binary approach.

In this approach, once the English binary has been created, its code is then modified to include other language-specific requirements and features. The customized binary is then localized into each target language.

Advantages of a Language-Dependent Binary

The advantage of using a language-dependent binary lies in its familiarity. This traditional approach is used in older versions of Windows, and is still widely used throughout the computer industry.

Disadvantages of a Language-Dependent Binary

One of the disadvantages is that a separate source tree, or at least a separate configuration of the build process, is required for each language. This goes against one of the fundamental rules of globalization: to create a single, worldwide binary.

Another problem is that changes to resources will require a full binary compilation. Imagine if you have a typo in your Japanese translation. Your functional binary would have to be recompiled, which would expose your application to the risk of potential defects.

Moreover, with this method valuable disk space is wasted when there are multiple copies of the core binary. For instance, imagine if a user is interested in having all language versions of your application. In this case, most of the core functions of your binaries, which are common to all languages, would be copied multiple times, needlessly consuming the user's disk space.

Furthermore, there is no real possibility of switching the UI language unless the existing process is ended and a new instance of your application is created in the desired language. The disadvantages of having language-dependent binaries are such that this approach is now considered obsolete.

Another method is to use one multilingual resource file.

Method 2: One Multilingual Resource File

The biggest disadvantage of the previous approach was the fact that multiple functional binaries were needed. The main idea behind this second approach is to separate out the resources from the source code, thus creating a resource-only DLL that contains all the localized resources for each targeted language. The functional part of the binary is language-neutral and thus common to all languages. Figure 6-3 illustrates the file distribution of such an approach.

Figure 6-3 File distribution for the multilingual resource file approach.
Figure 6-3 - File distribution for the multilingual resource file approach.

You can easily create multilingual resource files by using Microsoft Visual Studio's resource editor, in which multiple copies of the same resource identifier are created with different language IDs. The following example shows the string resource IDS_ENUMSTRTEST in both English and French within the same resource script (.rc) file:

// French (France) resources
#ifdef _WIN32
LANGUAGE LANG_FRENCH, SUBLANG_FRENCH
#pragma code_page(1252)
#endif //_WIN32

// String Table
STRINGTABLE DISCARDABLE
BEGIN
IDS_ENUMSTRTEST "Cette phrase est en français..."
END
#endif // French (France) resources

// English (United States) resources
#ifdef _WIN32
LANGUAGE LANG_ENGLISH, SUBLANG_ENGLISH_US
#pragma code_page(1252)
#endif _//WIN32

// String Table
STRINGTABLE DISCARDABLE
BEGIN
IDS_ENUMSTRTEST "This is an English string..."
END

#endif // English (United States) resources

To access the resources at run time, the resource DLL is loaded through the LoadLibraryEx application programming interface (API). You can then use EnumResourceLanguages to find the list of available languages for a given control or resource, and FindResourceEx to determine the location of the resource with the specified type, name, and language. Displaying a given language is then just a matter of selecting the right resources within the DLL. You can implement language switching by reloading the newly selected resources and refreshing the client area.

The Windows resource loader always defaults to the current thread locale for applications, which is inherited from the currently selected user locale. GetThreadLocale allows you to query the thread locale. In the multilingual resource file method, predefined resource-loading APIs (such as LoadIcon, LoadString, or LoadCursor) always return resources associated with the thread locale. For example, in the previous resource sample, the French resources cannot be loaded by using LoadString if the user locale is set to English (0x0409). However, once a thread locale is inherited-during thread creation-it is independent of the user locale. This means that the thread locale could, in this case, be changed to French (0x040C) by using SetThreadLocale and by calling LoadString to retrieve the French string. The following code sample demonstrates this principle:

// If the thread locale is English to start with
g_hInst = LoadLibraryEx(TEXT("intl_res.dll"),
   NULL, LOAD_LIBRARY_AS_DATAFILE);
LoadString(g_hInst, IDS_ENUMSTRTEST, g_szTemp, MAX_STR);

// g_szTemp would then point to the English resources,
// changing the thread locale to French.

// Always make sure that French is, in fact, one of
//    the valid languages returned by EnumResourceLanguages().

// Save a copy of the current thread locale to reset later.

Lcid = GetThreadLocale();
SetThreadLocale(MAKELCID(0x040c, SORT_DEFAULT));
LoadString(g_hInst, IDS_ENUMSTRTEST, g_szTemp, MAX_STR);
// g_szTemp would then point to the French resources.

The catch here is that if the thread locale is the same as the currently selected user locale, the system's resource loader will, by default, use the language ID 0 (neutral). If the desired resource is defined as a neutral language, then the data or resources will be returned. Otherwise, all of the language resources will be enumerated (in language ID order), and the first matching resource ID-regardless of its language-will be returned.

To clarify, consider an example using English (United States) and French (France) resources, in which the user locale is set to Japanese. Thus the original thread locale is Japanese (0x0411). The first call to LoadString will return English resources since the 0x0411 version of the string ID cannot be found, and since English (0x0409) is enumerated before French (0x040C). Now, if the user locale to start with is French, the original thread locale will be French (0x040C). Since the user locale and thread locale match, the system resource loader will use language ID 0, will start enumerating resources, and will once again return the English version!

The best way around this problem is to avoid changing the thread locale in the first place. Resources can be manually loaded from within multilingual resource files by making calls to FindResourceEx. Another workaround is to define the resources within an owner-defined language tag. In the example given earlier for .rc, the English resource section was defined as:

LANGUAGE LANG_ENGLISH, SUBLANG_ENGLISH_US

Defining the resources in this manner prevents the thread locale from being switched back to English (United States). However, defining the resources as:

LANGUAGE LANG_ENGLISH, SUBLANG_NEUTRAL

will guarantee the success of the SetThreadLocale call, since there is no matching user locale for this language ID and since the predefined resource loader APIs can still be used.

Advantages of the Multilingual Resource File

One of the advantages of this method is that you can switch UI languages on the fly by simply reloading a new set of resources from the language resource file and refreshing the UI area. Also, you can update resources without having to compile each of them. As a result, fixing translation defects or typos does not require that you compile the binary. The final advantage is that this method follows the underlying rule of globalization: that of creating a single, worldwide binary.

Disadvantages of the Multilingual Resource File

Despite the advantages, with this method it is difficult to update the resource binary to accommodate new languages. Suppose besides English, French, and Japanese, you also want to support German. A whole new resource binary would need to be provided to your existing users who might not necessarily need German.

Another disadvantage is that it is hard to install a subset of UI languages. For example, suppose 10 languages are available in the same file, but the user is only interested in a subset of four of these languages. The user is forced to waste memory and the disk space that the other six languages require because of the "one-size-fits-all" property of a multilingual resource. Another problem with a multilingual resource file is that changes for one language resource file may affect all languages and users.

Also, there is no straightforward way to implement direct resource-loading interfaces (such as LoadMenu, LoadString, and so on). On the whole, the big drawback of this approach resides in its lack of flexibility when it comes to adding, removing, or modifying the existing set of languages. To overcome this lack of flexibility, use satellite DLLs (Method 3), as discussed next.

Method 3: Satellite DLLs

This method is an expansion of the previous technique. Instead of a single DLL that contains all languages, a separate DLL is created for each language, compensating for the lack of flexibility in the previous method. This concept is somewhat like the Indian television network mentioned earlier that uses a single feed for a movie, but allows for subtitles in various languages. The satellite DLL approach is the one that Microsoft Office 2000, Microsoft Office XP, Microsoft Internet Explorer 5 and 6, Windows 2000, Windows XP, and the .NET Framework use. The technique is named differently depending on each product's implementation, but "satellite DLLs" is a commonly used, general name. Figure 6-4 illustrates the file distribution of such an approach.

Figure 6-4 File distribution for the satellite DLL approach.
Figure 6-4 - File distribution for the satellite DLL approach.

Just like in the multilingual resource file method, this technique requires that the resource DLL be loaded through the LoadLibraryEx API. This time, however, the appropriate language DLL must be selected. Typically it is safe to assume that the Windows UI language is the same as the user's preferred language and that the application can be started by loading this language resource. When the user selects another language, the current language can be freed and a new one loaded. Of course, this approach requires re-creating windows and initializing dialog boxes with the newly loaded resources.

Different variations of this technique exist. In the Windows XP MUI Pack, for example, the English resource files are embedded in the core binary, and satellite DLLs are offered for other languages.

Advantages of a Satellite DLL

Another advantage of satellite DLLs is that new language DLLs of an application can be released and shipped as they are made available, without having to ship the whole product. Also unique to this method is the ability to add a language DLL without affecting other languages.

A further benefit is that there is no need to worry about user locales, system locales, and thread locales. Finally, the satellite DLL technique can be used for any 32-bit version of Windows.

Disadvantages of the Satellite DLL

A disadvantage to this method is that you need to keep all language satellite DLLs synchronized. Doing this requires that setup and update procedures be adjusted in the development cycle to ensure that any update related to a change in resources will refresh resources for all installed languages.

Taking on the development model that employs satellite DLLs might also add some complexity to the developer's work. The following are some of the issues developers must address in this case:

  • Installation, configuration, and update routines of a product require more coding, since more files have to be tracked.
  • Resource-loading routines can no longer rely on a default behavior of the resource loader. Special code has to be written so that resource DLLs are located and loaded, and so that resources are retrieved. However, even though writing this code might require more time and resources to be invested in design, addressing these issues will yield a much better return on your investment in localizability.


How to Implement Your Own Satellite DLLs

Through a comparative study, the previous section has shown the undeniable advantages of creating satellite DLLs. This section will demonstrate how you can offer an MUI solution within your own applications by using satellite DLLs, whether you're dealing with Win32 applications, Web content, or the .NET Framework.

MUI in Win32 Applications

Assuming that you have already separated your resources from your core functional language binary, creating an MUI solution is as easy as following these basic steps:

  1. Detect the UI language of the operating system.
  2. Enumerate all languages for which you have a localized satellite DLL.
  3. Load the appropriate satellite DLL that matches the UI of the operating system.
  4. Allow the user to select a different language for your application, if you don't offer a localized solution for that particular language.

Before examining the technical implementation of MUI in Win32 applications, it's important to underscore that the only safe assumption for a fallback language is the UI language of the operating system. If you do not offer a localized solution for that language, then instead of defaulting to a given language (such as English), you will need to enable the user to select a language. For example, if the UI of the operating system is set to French, and yet you don't have localized files for French, you might be tempted to default to English resource files. This might be an accepted solution in Canada, where French and English coexist. But if a French user lives in Belgium, for example, odds are that his or her second language is Dutch rather than English.

Detecting the system's UI language is handled differently across various versions of Windows. In Windows Me, Windows 2000, and the Windows XP family of products, the new GetUserDefaultUILanguage API returns the language ID for the current UI language of the operating system. (In Windows MUI versions it is possible to have different users with different UI languages selected.) You can enumerate all available UI languages on the system by calling the EnumUILanguages API.

In Windows 95, Windows 98, and Windows 98 Second Edition, the UI language is stored in the registry at HKCU\Control Panel\Desktop\ResourceLocale. This key will return the language ID (LANGID) of the UI in hexadecimal (for example, 00000409 for English).

In Windows NT 3.5x, and Windows NT 4, because of the absence of a relevant API and consistent registry entries, the safest way to check the language of the operating system is to look at the version stamp of Ntdll.dll. The language of this DLL is the same as the language of the UI. The only exceptions to this approach are with Arabic, Hebrew, or Thai versions of Windows NT 4, where version stamping can help you detect an operating system enabled to support these languages. Thus the steps for checking the UI language of the operating system are:

  1. Load the Ntdll.dll file.
  2. Enumerate languages in the version-stamping resource.
  3. Note what the languages are for which the version stamping has a localized resource. If there is a language other than English (United States), the language of the operating system will be the same as this non-English language.
  4. Check to see if the version-stamping resource contains only English (United States). If so, you might still be dealing with enabled languages and should check against the active code page.

The following code sample demonstrates how detection of the system's UI language would work on Windows NT 3.5x and Windows NT 4 platforms:

HMODULE      hLib;
hLib = LoadLibrary(TEXT("ntdll.dll"));
// failed to load the file
if (hLib == NULL)
      return (FALSE);
// For East Asian countries, Ntdll.dll contains both English and
//    localized resources. Therefore, only take into consideration
//    the information for the non-English version.
EnumResourceLanguages(hLib, RT_VERSION, MAKEINTRESOURCE(1),
   EnumResLangProc, NULL);
FreeLibrary(hLib);
// If you only have English version stamping, you might still be
//    dealing with an enabled language (Arabic, Hebrew, or
//    Thai). Only true for Windows NT 4.
if (g_wLangID == US_LANG_ID
{
   UINT uiACP;
   uiACP = GetACP();
   switch (uiACP)
   {
      // Thai code page activated; this is a Thai-enabled system.
      case 874:
         g_wLangID = MAKELANGID(LANG_THAI, SUBLANG_DEFAULT);
         break;

      // Hebrew code page; this is a Hebrew-enabled system.
      case 1255:
         g_wLangID = MAKELANGID(LANG_HEBREW, SUBLANG_DEFAULT);
         break;

      // Arabic code page; this is an Arabic-enabled system.
      case 1256:
         g_wLangID = MAKELANGID(LANG_ARABIC,
            SUBLANG_ARABIC_SAUDI_ARABIA);
         break;

      default:
         break;
   }
}

// In our resource enumeration, we only keep non-English-
//    language stampings.
BOOL CALLBACK EnumResLangProc(HANDLE hModule, LPCTSTR lpszType,
   LPCTSTR lpszName, WORD wIDLanguage, LONG_PTR lParam)
{
   if (!lpszName)
      return FALSE;
   if (wIDLanguage != US_LANG_ID)
      g_wLangID = wIDLanguage;
   return TRUE;
}

All these techniques for detecting the system's UI language return a language ID composed of a primary language and a sublanguage. (For more information on primary languages and sublanguages, see Appendix D, "Table of Language Identifiers.") Two exceptions to this rule are Traditional Chinese (predominant in Taiwan) and Simplified Chinese (predominant in mainland China), which share the same primary language and yet have different localized versions. The same is true for Brazilian Portuguese and European Portuguese.

Now that you have identified the default UI language of the operating system, you need to load the satellite corresponding to that language. This process can be simplified by using generic naming conventions for your resource DLLs. If you only have one or two resource files, the suggested approach is to name them with the language ID, such as Myres409.dll for the 0x409 English (United States) satellite DLL. If your application has several resource files, a more practical solution is to create subdirectories named with the corresponding language IDs and place all satellite DLLs for the same language together. Figure 6-5 shows the file distribution for the GlobalDev sample application, which can be found in the Samples subdirectory on the companion CD.

Figure 6-5 File distribution for the GlobalDev sample application, where all resource files follow the convention GRes<LANGID>.dll.
Figure 6-5 - File distribution for the GlobalDev sample application, where all resource files follow the convention GRes<LANGID>.dll.

With this approach, by extracting the LANGID part of the file name and then using it to call GetLocaleInfo, you can find out the native name of each language resource file. The following code sample detects all language resource files currently available, makes sure that the languages are supported, and displays the list to a user:

int   nIndex = 0;
WIN32_FIND_DATAW wfd;
HANDLE hFindFile;
// The naming convention for resource DLLs is as follows:
//    GRes[LANGID].dll.
// Find all available resource DLLs in the current directory
//    but enumerate gres*.* files.
hFindFile = FindFirstFile(TEXT("gres*.*"), &wfd);

do{
      LANGID wFileLang;
      TCHAR szLangName[32];
      // Skip first four letters ("GRes")of file name and convert
      //    the rest to a LANGID.
      wFileLang = (LANGID) _tcstoul(wfd.cFileName+4, NULL, 16);

      // Since more languages might be offered than the user has
      //    support for, only list available
      //    and supported languages.
      if (IsValidLocale(wFileLang, LCID_INSTALLED))
      {
         // Get the native language name.
         GetLocaleInfo(MAKELCID(wFileLang, SORT_DEFAULT) ,
            LOCALE_SNATIVELANGNAME, szLangName, 32);
         // Add the new language to the list of UI languages.
         SendDlgItemMessage(hDlg, IDC_LANGUAGES, CB_INSERTSTRING,
            nIndex,
            (LPARAM) szLangName);
            nIndex++ ;
  }
      }
// Look for the following resource DLL:
while (FindNextFile(hFindFile, &wfd) );

In the previous code sample, adding new languages is transparent to the executable, since all you need to do is to put a new language resource DLL in the directory. The application enumerates the new file, finds its native language name, and adds this name to the list of supported languages. (See Figure 6-6.)

Figure 6-6 List of available UI languages created by the previous code sample.
Figure 6-6 - List of available UI languages created by the previous code sample.

Just as creating an MUI solution has its rewards for Win32 applications, offering this type of solution for Web content will have its own payoffs. Multilingual Web content will obviously be able to reach more people. Multinational corporations with operations around the world can more effectively accommodate employees who speak different languages when an Internet or intranet site offers multilingual capabilities. These are just a few instances for which an MUI solution can help, though many more examples could be given.

Multilingual Web Content

Glossary
  • Extensible Stylesheet Language (XSL): An XML language used for transforming XML documents into something that can be displayed, such as HTML.

Offering Web sites that provide content in the user's preferred language follows the same principles as providing multilingual content for Win32 applications:

  • Store your translatable resources in a database, in XML, or in resource files.
  • Reference string resources by variables.
  • Detect the display language at initialization time.
  • Load appropriate resources at display time.

Internet Explorer offers its own MUI solution (called "pluggable UI," which also uses satellite DLLs), where the browser UI language on English Windows 98, for example, can be set to Swedish. On Windows 2000 and Windows XP-which also offer their own MUI solution for the operating system UI-if the browser's MUI files are available, it tries to keep its UI language synchronized with the UI language of the operating system to avoid confusion.

Dynamic generation of language-specific Web interfaces from XML data is not hard-at least, it is not any harder than using XML to store UI data for a single language. The traditional model is to create an XML file containing all necessary data, bind the XML data to an HTML object on the page or apply an XSL transformation-and that's it! The client gets a nicely formatted page and your data, which is isolated from the code and UI, is easy to maintain. A multilingual scenario does not require much change to this traditional model. You will basically need to replace one single-language XML file with a set of language-specific files, which are loaded according to the settings of the client. An important step in building a UI according to the user's preferences is getting those preferences in the first place. There are several methods you can use:

  • Use window.navigator.browserLanguage to check the language of the browser (for the UI language) and use window.navigator.- userLanguage to checkthe language of the user (for cultural preferences). Query the values in a client-side script and post them to the server for a server-side UI construction.
  • Provide the user with a list of available languages and send (post) user feedback to the server. This solution can be combined with the next one.
  • Query the user's profile. You can use this method if the user is authenticated (for example, with Microsoft .NET Passport).
  • Query the value of HTTP_ACCEPT_LANGUAGE to find out what languages the browser accepts. If the browser accepts multiple languages, you can choose whatever language your resources happen to be in, obviously selecting from the preferred languages first.

The advantage of the last two methods is that you can use them at a session's initialization step-for example, in the Session_OnStart procedure of Global.asa. However, these two solutions might be less flexible than the first method, in which you query values in a client-side script. In effect, the last two methods are equivalent to making an educated guess about the user's preferred language. Suppose the browser language is set to Finnish and your Web pages are not available in that language. You might default to English, not realizing that the user's second language is Russian, not English. Furthermore, there is not much advantage to the last two methods if the Session object is disabled in the first place.

Rather than selecting one particular algorithm as the preferred solution for all scenarios, it is wiser to base your solution upon the specific business case at hand. As you will see in the the sample code that follows, the decision on which language to use to display content is isolated within the GetSessionLanguage() function. In order to make GetSessionLanguage() universal, the function returns the language ID. However, it would be better to set the Session's UI language variable. As a general rule, you should set a language for each user's session. This way you allow different languages for different users, and you can confine your decision about which UI language to a single instance-within the initialization code. Some applications might avoid using Session states, in which case a language ID returned as a string comes in handy. Note also that GetSessionLanguage cannot be made completely universal; it will depend on the naming conventions you adopt for the XML files in your product, the set of languages you have localized into, and the default language of your application. The following code illustrates one mechanism possible for dynamic construction of a language-specific UI. To make things more general, it is not specified whether the code is executed in a Global.asa initialization of a session or later. Error handling was also removed from the code, but you should be able to handle errors that might arise, such as when the XML file for a particular language cannot be found.

<%@LANGUAGE = "VBScript" %>

<%

' Function GetSessionLanguage() is omitted for simplicity. It
'    would be application-specific anyway. You can find a sample of
'    this type of function in the Samples\MultiLanguageWeb\
'    BrowserSniff subdirectory on the companion CD. The function
'    implements language detection based on the value of
'    HTTP_ACCEPT_LANGUAGE.
' It is assumed that the function returns a language ID string
'    that is never empty.

' The default language of the site is returned if the language
'    of the user's choice is not available.
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
' InitializeUI() detects the language to be loaded,
'    stores the language ID in a session variable,
'    loads the proper resource XML file and
'       language-independent XSL template,
'    generates the UI, and writes it to the client.
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Sub InitializeUI()
'Returns a non-empty string
Session("ui_language") = GetSessionLanguage()

' If your languages canot be presented in one code page, you
'    can set the Session.Codepage and Response.Charset values
'    here, based on the language.
' But always handling text in UTF-8 might be a better solution.

Session.Codepage = 65001
Response.Charset = 65001

Set Session("xmlData")  = Server.CreateObject("Msxml.DOMDocument")
Set Session("xmlStyle") = Server.CreateObject("Msxml.DOMDocument")

Session("xmlData").load(
   Server.MapPath("xml/"&Session("ui_language")&"/uiRes.xml"))

Session("xmlStyle").load(Server.MapPath("xml/uiTemplate.xsl"))
Response.Write(
   Session("xmlData").transformNode(Session("xmlStyle")))

End Sub
...
%>

The XML file can contain all the information needed for on-the-fly generation of localized content. If you need to apply directionality to your localized pages, use language-specific graphics or URLs. In the previous code sample, if English, French, German, and Japanese resources are defined, the structure of the "resource" tree looks like this. (See Figure 6-7.)

The assumption is that the same information is kept in each language version of the XML resources, thus preserving the same schema among the languages. With this model you can use the same XSL transformation on all of the files, which will simplify your code. Adding new language resources becomes easy as well.

When dealing with MUI in the context of the .NET Framework, you will need to consider the role of the CurrentUICulture property, as well as how resources are generated and loaded. The following sections explore these topics.

Figure 6-7 Structure of resource tree.
Figure 6-7 - Structure of resource tree.

Resource Handling in the .NET Framework

Glossary
  • Common Language Runtime (CLR): A very important part of the .NET Framework. At the base level, it is the infrastructure that executes applications, and allows them to interact with the other parts of the .NET Frame work. It also provides important capabilities in optimizing, securing, and providing many robust capabilities such as application deployment and side-by-side execution.
  • Hub and spoke model: The hub is the main assembly that contains the nonlocalizable executable code and the resources for a single culture, called the "neutral" or "default culture." The default culture is the fallback culture for the application. Each spoke connects to a satellite assembly that contains the resources for a single culture, but does not contain any code.
  • Overload: The concept of defining a procedure in multiple versions, using the same name but different argument lists. The purpose of an overload is to define several closely related versions of a procedure without having to differentiate them by name. You do this by varying the argument list.

The Microsoft .NET Framework offers a whole new approach to the way resource files are created and loaded. Through the support offered by the CLR, you can provide an MUI solution in your .NET applications much more easily. Moreover, with the .NET Framework you can implement MUI in a way that is practically transparent. The CurrentUICulture property is a vital part of resource handling.

CurrentUICulture Property

In the new naming conventions of the .NET Framework, the CurrentUICulture property of the CultureInfo class (from the System.Globalization namespace) is a per-thread setting. You can retrieve CurrentUICulture by querying the Thread.CurrentUICulture property, and you can change CurrentUICulture by setting Thread.CurrentUICulture. Thread.CurrentUICulture is used by the ResourceManager class to look up culture-specific resources at run time.

The CultureInfo class specifies a unique name for each culture based on the Request for Comments (RFC) 1766 standard. This standard uses the format <languagecode2>-<country/regioncode2>, where <languagecode2> is a lowercase two-letter culture code associated with a language, derived from International Organization for Standardization (ISO) 639-1, and where <country/regioncode2> is an uppercase two-letter subculture code derived from ISO 3166. As discussed in Chapter 4, the invariant culture-which is not associated with any particular language, country, or region-is the root of all cultures. A neutral culture is associated with a language and can be used for resources. (The neutral culture is the equivalent of the primary language ID in the Win32 programming paradigm.) For instance, "fr" is a neutral culture indicating that the language you are dealing with is French, but the neutral culture makes no provision about the actual location (country or region) in which French is being used (France, Belgium, Canada, and so on).

A specific culture is associated with both a language and a region and provides formatting-specific information for that particular location. So, for example, "fr-FR" indicates that you are dealing with the French language and with regional settings that are appropriate for France. (See Chapter 4.) Figure 6-8 summarizes this hierarchy, using German and English as examples. Within the neutral culture of German, for instance, are German in Austria, Switzerland, Germany, Liechtenstein, and Luxembourg. The neutral cultures, German and English, are both part of the invariant culture.

Figure 6-8 Hierarchy of invariant, neutral, and specific cultures.
Figure 6-8 - Hierarchy of invariant, neutral, and specific cultures.

The CurrentUICulture property is set by the framework when the application starts so that it matches the user's default UI language. Alternatively, you can set CurrentUICulture explicitly in your application's code. The following code example sets the CurrentUICulture property to the neutral culture "de" for German.

Thread.CurrentThread.CurrentUICulture = new CultureInfo("de");

You can also set the CurrentUICulture property to a specific culture. The following code example sets the CurrentUICulture property to the specific culture "de-DE" for German in Germany:

Thread.CurrentThread.CurrentUICulture = new CultureInfo("de-DE");

As you have seen, the CurrentUICulture property can help you handle resources in a culture-specific manner. The following sections will show how resources are created and loaded.

The Resource Generation Process in the .NET Framework

As in the Win32 programming model, in the .NET Framework your resources should be separated from the rest of your code. Resource files in .NET are created within .resx files; these files consist of XML-based entries that specify objects and strings within XML tags. One advantage of a .resx file is that when opened with a text editor (such as Notepad or Microsoft Word), it can be written to, parsed, and manipulated.

When viewing a .resx file, you can actually see the binary form of an embedded object (a picture, for example) when this binary information is a part of the resource manifest. Apart from this binary information, a .resx file is completely readable and maintainable. A .resx file contains a standard set of header information, which describes the format of the resource entries and specifies the versioning information for the XML that is used to parse the data. Following the header information, each entry is described as a name/value pair. A name/value pair in the .resx format is wrapped in XML code, which describes string or object values. When a string is added to a .resx file, the name of the string is embedded in a <data> tag, and the value is enclosed in a <value> tag.

The resource generation process is straightforward and includes three steps (although the first and second steps are optional):

  1. A .resx file that contains an XML-based description of the resources is compiled into a .resources file by using the ResGen tool (within the Microsoft .NET Framework SDK available at http://msdn.microsoft.com). The .resources file that is created contains binary resources. The ResGen tool can also perform the reverse transformation, generating a .resx file from a .resources file.
  2. Another optional step is to create a text file that contains resources such as strings and to transform this text file into a .resx file by using the ResXGen tool provided in the Microsoft .NET Framework SDK. Note that the ResXGen tool is different from the ResGen tool mentioned in step 1. Figure 6-9 shows the .NET resource generation process used in the .NET Framework.
    3. Figure 6-9 The .NET resource generation process.
    Figure 6-9 - The .NET resource generation process.
  3. The .resources file is embedded into an assembly by using either the assembly linker tool or a language compiler, such as the C# compiler (CSC).

Resource Loading in the .NET Framework

The ResourceManager class provides convenient access to culturally appropriate resources at run time. This classmanages multiple resources from a common source that has a particular root name, like the hierarchy shown earlier in Figure 6-8. ResourceManager objects provide fallback resource lookup to region-independent and neutral cultures when specific localized resources are not provided. Resource-loading requests are based on the culture that is associated with the current thread. This culture can be set through the Thread.CurrentThread.CurrentUICulture property.

The ResourceManager class provides two methods, GetString and GetObject, which enable an application to load either a string resource or any object (that can be serialized) from an assembly. Both the GetString and the GetObject methods support two types of overloads:

  • An overload where only the name of the resource is used as a parameter.
  • An overload where, besides the name of the resource, an instance of the CultureInfo class must be provided. The CultureInfo object represents the culture for which the resource is localized.

If the resource is not localized for the culture that is requested (either explicitly through the second overload or implicitly, by using CurrentUICulture), the lookup will fall back using the culture's Parent property, stopping after looking in the default resources.

When you package your application's resources, you must name them using the resource-naming conventions that the CLR expects. The run time identifies a resource by its culture signature, or name, as discussed in "CurrentUICulture " earlier in this chapter. The .NET Framework uses a hub and spoke model to package and deploy resources. This model requires that you place resources in specific locations, so that they can be easily located and used. If you do not compile and name resources as expected, or if you do not place them in the correct locations, the CLR will not be able to locate them.

If your application includes resources for specific cultures (such as "de-DE"), place each specific culture in its own directory. Do not place specific cultures in subdirectories of their respective neutral culture's directory. (For example, do not put the "de-DE" resources in the "de" folder.) If you do so, the application will not be able to find the localized resource.

The hub and spoke model for packaging and deploying resources uses a fallback process to locate appropriate resources. If an application user requests a ResourceSet that is unavailable, the CLR searches the hierarchy of cultures looking for an appropriate fallback resource that most closely matches the user's request, and raises an exception only as a last resort. At each level of the hierarchy, if an appropriate resource is found, the run time uses it. If the resource is not found, the search continues at the next level.

For instance, if an application is localized in French using the "fr-FR" culture for its resources satellite assembly, and if the application is looking for a string that cannot be found in "fr-FR," it will then fall back to the French culture (if available) and try to find the string in the "fr" satellite assembly. If the string is not available at that level, it will then fall back to the default resources in the main assembly. Finally, if the string resource is not found at that level, an exception will be raised. Figure 6-10 illustrates the fallback process.

The final MUI solution to consider applies to console or text-mode applications. Again, you must think about how resources are handled and loaded within this particular context.

Figure 6-10 The .NET resource fallback mechanism.
Figure 6-10 - The .NET resource fallback mechanism.

Resource Handling in Console Applications

A Win32-based console application finds and loads resources in the same manner as any Win32 code. However, when handling resources in console (or text-mode) code, you must take into account some specifics of the Windows console subsystem. An obvious requirement for any code is that it produce a UI that is easily readable. Additionally, the UI language and settings must be based on the user's preferences. Unfortunately, there can be no warranty against the situation when the UI language cannot be represented in the console output code page. If this happens, loading and displaying resources in the language of the user's UI does no good-the console will not be able to display the output.

The easiest way to prevent the UI from breaking is to make resource loading dependent upon the console code page. Picking which language resource files to load can be tricky, though. One possible method, which can be applied to many scenarios, involves the following steps:

  1. Get the user's UI language as described in "MUI in Win32 Applications" earlier in this chapter.
  2. Get the code page of the UI language by generating an LCID for the language and calling GetLocaleInfo.
  3. Get the console output code page with GetConsoleOutputCP.
  4. If the code pages found in step 2 match the console code page, and the UI language is not Hebrew, Arabic, or another language that uses complex scripts, load the resources that match the user's UI language. Otherwise, load English resources.

The advantage of loading English resources is that they can always be displayed; but you should be aware that, by doing so, you are losing cultural accuracy. You can also select another approach, such as setting the language that you are going to load to match the console code page, regardless of the UI language. The catch here is that the UI language is a per-user setting, and the console code page follows the system locale; the user cannot always change the system locale. Remember also that setting the ThreadLocale affects locale-sensitive operations such as date formatting, and should be used cautiously. If you use multilingual resource sections and change the thread-locale value to select the language to load, do it only for the time resources are loaded, and restore the original value after that. Console applications depend heavily on text resources, often building the output at run time from string table entries, using the functions of the printf family for run-time parameter substitution. In many cases, this practice makes localizability of the code difficult. When multiple parameters have to be substituted at run time, their relative positions might differ from one language to another, but the order of the printf arguments cannot be changed without code modifications. Using message tables together with FormatMessage API eliminates this problem.

Message tables are a Win32 resource that uses sequential numbers rather than escape letters to mark replacement parameters, making it convenient to store messages that contain several (up to 99) replacement parameters. The Format-Message API function will substitute variables according to each placemarker's numeric label and not according to its position in the string. Localizers can freely change a string's word order, and FormatMessage will still return correct results.

Message tables are defined using message compiler (.mc) files and are compiled into resource files using the message compiler, mc.exe. The format of the message table is designed so that multilingual error messages are easier to interpret. To accomplish this, the message table includes flags for error severity and for the facility that caused an error. For instance, the message text file contains a header that defines names and language identifiers used by the message definitions in the body of the file. The header can contain the following statements:

  • MessageIdTypedef = [type ]
  • SeverityNames = (name=number[:name])
  • FacilityNames = (name=number[:name])
  • LanguageNames = (name=number:file name)
  • OutputBase = {number}

These flags are optional and can be omitted if the message table is used to store ordinary text resources. The following code sample illustrates how multilingual message tables with no "redundant flags" are created:

// SAMPLE.MC
LanguageNames = (English=0x409:MSG00409)
LanguageNames =(German=0x407:MSG00407)
MessageId=1 SymbolicName=IDS_NOFILE
Language=English
Cannot open file %1
Language=German
Die Datei %1 kann nicht geöffnet werden.

MessageId=2 SymbolicName=IDS_OTHERIMAGE
Language=English
%1 is a %2 image.

Language=German
%2-Abbild ein %1 ist.

The following code shows how the previous message table can be used in a program. The language ID here is set to NULL. The resources will be loaded according to the ThreadLocale settings.

wchar_t       lpBuf[60];
LPVOID        lppArgs[10];
DWORD len = FormatMessage(
   FORMAT_MESSAGE_FROM_HMODULE | FORMAT_MESSAGE_ARGUMENT_ARRAY,
   NULL,     // message source - 0 stands for current module
   idMsg,    // ID of the message to be loaded
   NULL,     // Language ID
   lpBuf,    // Destination buffer
   sizeof(lpBuf)/sizeof(TCHAR),
   lppArgs   // Array of message inserts - list of strings
);
Important
Most behavior of C run-time functions is based on their own run-time locale, not on the Windows system or user locales. (For more information, see the MSDN documentation on the setlocale API at http://msdn.microsoft.com.)

Also, most C run-time text display and input APIs-depending on the C run-time implementation-are still code-page-dependent. It's strongly suggested that you use Windows console APIs for display and input in order to take advantage of an interface that is fully Unicode-based.


Summary

The MUI Pack is a set of language-specific resource files that can be added to the English version of Windows 2000 and Windows XP. MUI allows the UI language of the operating system to be changed to one of many supported languages. One of the benefits of MUI is that one user can see system menus, dialog boxes, and other text in one language, while another user logging onto the same system can see the same text in a different language. A Windows system with MUI set to a particular UI language will largely look and behave like the localized version, with some exceptions. Since MUI runs on top of the English version of Windows, from a feature and architectural point of view, localized versions of Windows are the same as the MUI version on English Windows XP.

There are three principal methods (with many variations thereof) available for offering an MUI solution for your own applications. The first method is to compile one binary (with built-in resources) for each language. The second method is to have one language-neutral core binary with one multilanguage resource DLL. The last method is to have one language-neutral core binary with one set of resource DLLs-a satellite DLL-for each language. While each method has its advantages and disadvantages, the last method is generally preferred.

When offering your own MUI solution, detect the UI language of the operating system; enumerate all languages for which you have a localized satellite DLL; load the appropriate satellite DLL that matches the UI of the operating system; and allow the user to select a different language for your application, if you don't offer a localized solution for that particular language. By adding MUI capabilities to your applications, you will have completed the last step in globalizing your software.


Part III: Localizability

Part II, "Globalization," was entirely dedicated to creating a fully globalized software solution, which includes designing and implementing source code so that it can accommodate any local market (locale), script, or language. You might decide to leave your user interface in its native language, but to enable its functionality for multilingual usage. If this is the case, Part II offers valuable guidance. In contrast, Part III,"Localizability," is dedicated to creating localizable software—in other words, designing software code and resources such that resources can be localized for any locale and language without breaking functionality and without requiring changes to code. Therefore, if you are not planning to localize your software, which in today's global economy is most unlikely, you can safely skip this stage all together. Both globalization and localizability are concerned with providing unified and equal support for all locales, scripts, and languages.

Localizability can be divided into software localizability and content localizability. Chapter 7, "Software Localizability Guidelines," addresses the first of these components, by offering recommendations on working with resource repositories, as well as showing you what to avoid from a coding standpoint. You will see reasons why you should not hard-code localizable resources, tips on how to handle strings, ways to minimize resizing and repositioning of the user interface, and guidelines regarding icons and images.

Chapter 8, "Mirroring," discusses a specific technology for handling right-to-left languages—such as Arabic and Hebrew. Within the context of Win32 applications, you will see ways to enable mirroring in code. For Web pages, you will see methods and code you can use for rendering right-to-left layout of text, of dynamic elements such as tables and cells, and of static elements such as directional images. Finally, for the .NET Framework, you will learn techniques for mirroring Web Forms, Microsoft Windows Forms, and message boxes. Chapter 9, "Content Localizability Guidelines," examines practices for making printed and online documentation, Help files, and marketing material simple, precise, and culture-aware. The chapter also demonstrates how to enable content recycling to decrease localization costs and how to design your Help system so that it has global functionality.


Chapter 7: Software Localizability Guidelines

Most localization teams encounter a variety of localization difficulties. These challenges culminate in a product that cannot be completely or properly translated; this, in turn, results in both a higher than necessary cost for localizing a product and a product that is of lower quality, less usable, less stable, or harder to support. Often these issues can be tackled at the source by designing and writing software that follows basic localizability guidelines. Adhering to these guidelines increases the quality, accuracy, and user-friendliness of a product's international versions. In some cases it even improves the original-language product. Moreover, following these practices can dramatically decrease localization costs and frustrations.

For example, suppose it takes one minute to localize a text string. Using an arbitrary wage of $25 USD per hour, this costs about 42 cents. If the product is localized into 20 languages, this one string costs $8.40. If this string is difficult to localize, either for linguistic reasons or because the user interface (UI) is badly designed, suppose it now takes five minutes to localize the string. If this is the case, this same string, for all 20 languages, costs $42. If 25,000 strings are difficult to localize, it costs $1,050,000. Thus what should have cost $210,000 would cost $840,000 more because developers and designers did not follow good localizability practices.

Some of these additional costs are incurred because localizability bugs aren't found until late in the development process-after a program has been translated. Unless some type of pseudo-localization is done, it usually takes someone who speaks the target language to find these bugs. Also, any changes made to fix a localization bug for one language can have an impact on other languages.

On the positive side, despite the fact that localizability bugs are discovered late, they are generally easy to fix. In contrast, globalization bugs are easier to find earlier in the development process, since they can be tested for as soon as a module is written. However, globalization flaws might require design changes and more time to address, for example, porting an existing application to Unicode. Most globalization bugs are found and fixed before they can affect the localization of the program. To help detect localizability bugs sooner, companies like Microsoft are adding new practices such as pseudo-localization. (For more infor-mation on pseudo-localization, see Chapter 12, "Testing Localizability with Pseudo-Localization.")

This chapter will discuss software localizability in terms of design, coding, and content practices that make localization more efficient. You will see tips on string handling, ways to minimize resizing of the UI, techniques for making UI controls localizable, and recommendations for using images and icons.


Isolating Localizable Resources

Glossary
  • Localizable resource: Any element of a program's UI that requires translation or modification for different languages. These elements are either UI resources or resources that need to be modified for the adaptation of a localized product (font information, locale information, folder names, account names, and so on).
  • Hard-coding: 1) Putting string or character literals in the main body of code, such as the .C files or the .H files, instead of in external resource files. 2) Basing numeric constants on the assumed length of a string or having any assumptions about language- or culture-specific matters fixed in the code (such as length of strings, formats of dates, and so on).

Once you know what you are looking for, it is easy to identify source code that isn't properly set up for localization. The most extreme cases are populated with hard-coded strings, constants, and characters. Files containing hard-coded localizable elements are the biggest nightmare to deal with. Translators cannot easily translate the source-code files, especially if the code is continually evolving. Many translators are not programmers and might delete important details, such as closing quotation marks or semicolons, or they might translate programming keywords as well as strings. Then developers must take time to clean up these files so that they will compile correctly. Hard-coded elements are difficult to find because they do not show up until the software is localized. Figures 7-1 and 7-2 show some common problems that occur during localization.

Figure 7.1 The string Unknown Modem was hard-coded and thus went untranslated.
Figure 7.1 - The string "Unknown Modem" was hard-coded and thus went untranslated.
Figure 7.2 The string is a text resource without a period at the end of sentence, and the period was hard-coded.
Figure 7.2 - The string is a text resource without a period at the end of sentence, and the period was hard-coded.

Any translator would assume that the lack of punctuation in the previous figure is a mistake and would end the translated sentence with a period. Thus the localized version ends up with two periods.

Separating all localizable items into one or more files makes localization much easier to manage. Table 7-1 shows the major program elements that require localization.

Table 7-1 Major program elements requiring localization.
Localizable Items
Menus | Messages | Dialog boxes
Prompts | Images | Sounds
Toolbars | Status bars | Constants

One approach would be to create a complex process that strips out all localizable items from source files and that reincorporates these items once they have been translated. That way, you could declare strings in the same area as the code that uses them. However, designing and implementing such a scheme hardly seems worth the effort when better localization methods are available. An alternative approach would be to place all strings and constants in header files, but then you would have to recompile any source files that included translated headers.

By far the simplest and most straightforward method of dealing with localizable resources is to put everything in some type of resource repository like Windows resource files, .NET assemblies files, or a database when dealing with Web content. Resource repositories, which will be described in more detail later in this section, tend to be easy to edit, and they eliminate the need for recompiling source code. Because of these advantages, developers have used resource repositories as a spot to store resources referenced in one or more places in their code. This actually adds complexity to what from a localization perspective should be a resource file containing only resources that need localization. Resource repositories have the following resources:

  • UI resources. These can be localized in an arbitrary way without any loss of functionality.
  • Resources used for the adaptation of a product (font and locale information, folder names, account names, and so on). These strings need to be localized according to certain rules; incorrect localization has functional impact. If possible, these resources should be queried from the supporting platform and not stored within the application.
  • Debug resources. These shouldn't be localized, but there is no loss of functionality if they are.
  • Functional resources (registry keys, function calls, strings communicated between components, and so on). These strings can't be localized without loss of functionality.

To complicate things further, there are also resources that don't fall clearly under just one category (for example, functional text that is being displayed in the UI, or UI strings that have to be localized in a certain way so that they don't break the product). This is a huge challenge for localization because there is no way to tell what category a resource belongs to without looking at the code. Since localizers generally don't have access to the code (and many don't understand the code anyhow), they are depending on their own judgment. As a result, they introduce many defects that are costly to detect and address.

The goal is to separate the first two categories (UI resources and resources used for product adaptation) from the last two categories (debug resources and functional resources) and to make sure that resources clearly belong to one category only. Since localizers usually don't have access to registry keys, it is best if you keep all text out of them and store the text strings in your localizable resource. Keeping text strings out of registry keys also makes it easier to develop Multilingual User Interface (MUI) solutions. (For more information, see Chapter 6, "Multilingual User Interface [MUI].") To further reduce complexity and to increase flexibility of the code, you should keep the category of resources used for the adaptation of a product as small as possible. For example, instead of using multiple instances of strings that need to be localized identically (like folder names and account names), refer to one instance of this string only.

Although moving localizable items like strings and icons into resource repositories is a good localizability practice, it is important to emphasize that just like anything that is good, sometimes more is way too much. Just because one chocolate bar tastes good, eating 10 at one sitting can leave you with some undesirable consequences. This goes for resource files also. As mentioned earlier, there are four distinct localizable categories for resources. One of the most counterproductive practices is putting all of your resources (including items that do not need to be localized) together in one place, rather than separating them into different resource files by categories. This practice can confuse localizers, causing two major problems to occur:

  • The unnecessary translation of resources in general (which wastes money that could be used elsewhere).
  • The inadvertent translation of resource names that are specific or necessary to your application. (One example includes a command string like "Open" that your code does not recognize anymore because it has been translated to "Ouvrir" in French. Another example is translating the name of a mutex or event that an application is checking for in the source language.)
  • Also, you should not expect services and system components to always be in English, since they might have been translated for localized versions. For instance, the following code will fail when run on a localized version because all values in bold are translated on localized systems!
// user is system, which means cannot log on
if ( (_tcscmp(lpBuffer,TEXT("SYSTEM")) == 0) ||
     (_tcscmp(lpBuffer,TEXT("LOCAL SERVICE")) == 0) ||
     (_tcscmp(lpBuffer,TEXT("NETWORK SERVICE")) == 0)

A better way to handle resources that come from localizable platforms is to use application programming interface (API) calls to get the system's localized values. The following example uses SHGetFolderPath, which will return the correct localized version of the path from one localized system to another localized system. (For information on the different Shell calls, see the "Shell Programmers Guide" at http://msdn.microsoft.com.)

TCHAR szPath[MAX_PATH];
SHGetFolderPath(NULL, CSIDL_PERSONAL, NULL, 0, szPath);

A good practice is to put resources that do not need localization into header files (like .H files for the C programming language). This gives you all the necessary flexibility of a resource file, and yet these resources will never get confused with localizable resources. Also, if the technology you are working with does not have some way to separate localizable resources, use whatever feature it does have that will ease the localizer's job. Always keep in mind that you should group nonlocalizable items separately from localizable items.

Another good practice when working with resources is to not change resource identifiers. Most resource repository systems have some index the code can use to access the string or other item needed for creating the UI. It is the use of this index that makes resource files great choices for localization. By merely changing the repository from one filled with the original language to one that contains the new language, the index now points to the correct language.

This action is not without peril. Suppose that during an update process you decide that you need to change a resource identifier (leaving the text in the resource unchanged). By doing this, you have now made your translated resourcerepository unusable for your update because your code expects the text to be located somewhere else in the repository. Consequently, if you do nothing, you might get the wrong text. To avoid this, you would have to go in and change your translated resource by at least moving the old translation to the new location or by doing the translation all over again. The latter is contrary to the guiding principle of keeping required translations to a minimum. (For more technical information on resource handling for Microsoft Win32, the Microsoft .NET Framework, and the Web, see Chapter 6.)

When dealing with resource repositories, you will save a lot of money if you remove unused strings and dialog boxes before having the resources localized. Translators and localizers have no way of knowing which of your strings are used and which are not. Thus all they do is what you tell them (such as to translate a particular resource file). For example, suppose one out of every 50 strings in your resources is not used. Referring back to the 25,000-string project, that means there are 500 strings in the resources that do not need to be translated. If you have 20 languages costing 42 cents per string to localize, those 500 strings end up using $4,200 of your budget-money that could have been allocated elsewhere.

You have seen the efficiency of using resource repositories to isolate localizable resources. Now take a look at some recommended practices for handling strings, as well as how you can implement various mechanisms to deal with the order of variables.


String Handling

Once you have chosen a resource store, you can move on to the next area that causes a lot of localizability nightmares-the way strings are created for display in the UI or for placement in a persistent storage. Many a translator has spent sleepless nights trying to decipher what a creative developer is attempting to convey to the user. There are at least as many different ways to create strings of text as there are developers, and thus it would be impossible to include all of them here. However, this section will focus on five major areas to watch out for in relation to strings.

Avoid Run-Time Composite Strings

When creating strings for output, well-intentioned programmers use a coding trick that has been passed down from generation to generation of developers as a good coding practice. That trick is using text strings with variables that get composed at the application's run time. Thus you see code as shown here that is written to produce the following English text:

"Are you sure you want to delete the file?";
"Are you sure you want to delete the directory?";
"Are you sure you want to delete the subdirectory?";
char szString[] = "Are you sure you want to delete the ";
char szFinalString[cbMaxSz]= szString + szDelObject + "?";

Note that szDelObject is a variable that can contain either a file, directory, or subdirectory. This practice works well for the programmer, but all the translator sees in the resource file is "Are you sure you want to delete the "; the translator has no idea what is being deleted. The value of the variable often determines whether a different syntax or definite article needs to be used, depending on the item to be deleted. The reason is that in some languages nouns have gender. For instance, in German nouns can be masculine, feminine, or neuter. In Romance languages such as French or Portuguese, nouns are either masculine or feminine. Depending on the gender of the noun in such languages, the definite article that precedes the noun will vary. For example, French uses "le" in front of masculine nouns (such as "le fils" for "the son") and "la" in front of feminine nouns (such as "la fille" for "the daughter"). Thus for the previous code, a French translator would not know whether to use "le" or "la," because the actual value of the variable-in this case, the noun-is not specified.

The best way to avoid this ambiguity is to write out each sentence completely, if possible. This way the translator knows the context of the sentence and can translate it correctly. Thus in your resource store you would have:

Del_File = "Are you sure you want to delete the file?";
Del_Dir = "Are you sure you want to delete the directory?";
Del SubD = "Are you sure you want to delete the subdirectory?";

Another potential shortcut involves declaring a single string, such as "none," "blue," or "first," and displaying it in a number of different contexts-on a menu, in a dialog box, and perhaps in several messages. The problem with using "all-purpose" strings is that in European languages, adjectives (and some nouns) have anywhere from 4 to 14 different forms (for example, masculine, feminine, and neuter singular; and masculine, feminine, and neuter plural) that must match the nouns they modify. For instance, in Spanish the word "first" can be conveyed by the words "primero," "primera," "primer," "primeros," and "primeras," depending on the gender and number of the noun or the particular sentence structure. A single string displayed in different contexts will be correct in gender and number in some cases but incorrect in others. The user will consider such a translation amateurish. Again, the way to handle this problem is to have a separate string for the same word used in a different context, such as the following strings:

    Menu_open = "open"
    Dialog_open = "open"
    Button_open = "open"

When Variables Are Necessary, Use Unique Names

Although translators would love to have strings with no variables as well as strings that are repeated for usage in different contexts, there are times when it is necessary to use variables. When there is only one variable in a string, like in the example that involved deleting a file, directory, or subdirectory, it is best to use a function that allows you to at least put a placeholder in the text. Thus the earlier example would become:

Del_File = "Are you sure you want to delete the %s";

To help the translators, document all the values that the variable can be. This way they will know what goes in the variable and will be able to translate it using the correct syntax and grammar for the target language.

Because word order can vary significantly from language to language, using identical placeholders can cause problems if you have a string that has two or more variables. For example, suppose you have the following string:

Memory_Error = "Not enough memory to %s the file %s.";

The first %s is a function like open, close, or save and the second %s is the name of the file you are trying to open, close, or save. Thus one of the possible sentences generated with this string might be:

"Not enough memory to open the file filename1."

Suppose you plan to translate this text into Swedish and Finnish. The Swedish translation causes no problem because its translation is:

"Det finns inte tillräckligt med minne för att öppna filen
FileName1."

Notice that the command "open" ("öppna") comes before the file name. Now look at the Finnish translation:

"Liian vähän muistia tiedoston FileName1 avaamiseen."

Notice here that the command "open" ("avaamiseen") comes after the file name; but in most programming languages if you used %sfor both your variables (as in the earlier example), your program string would actually be created as:

"Liian vähän muistia tiedoston avaamiseen FileName1."

This is because the variable in the original call to output this string was in the order of command first, file name second. Therefore, it would be better if you had some way to show variable order, as in this string in English:

"Not enough memory to %1 the file %2."

The Swedish and Finnish translation, respectively, would be:

"Det finns inte tillräckligt med minne för att %1 filen %2."
"Liian vähän muistia tiedoston %2 %1."

The following shows the Win32 and .NET Framework mechanisms created to handle variable order.

FormatMessage

Glossary
  • Message table: A Win32 resource that uses sequential numbers rather than escape letters to mark replacement parameters, making it convenient to store alert messages and error messages that contain several replacement parameters.

Win32 supports two resource types for storing strings: string tables and message tables. (For more information on message tables, see Chapter 6.) String tables make sense for short strings and for strings containing only one replacement parameter; message tables are more convenient for alert and error messages that contain more than one replacement parameter. (Message tables support up to 99 parameters.) The FormatMessage API will substitute variables according to each placemarker's numeric label and not according to the label's position in the string. Localizers can freely change a string's word order and FormatMessage will still return correct results. The file format of the message table is not complicated; you can create message tables with a simple text editor. The following code appears in a message table that contains English and German translations of the same strings. The German translation of IDS_OTHERIMAGE reverses the positions of the replacement parameters.

// Sample.mc
LanguageNames=(German=2:msg00002)

MessageId=1 SymbolicName=IDS_NOFILE

Language=English
Cannot open file %1.
Language=German
Die Datei %1 kann nicht geöffnet werden.

MessageId=2 SymbolicName=IDS_OTHERIMAGE
Language=English
%1 is a %2 image.

Language=German
%2-Abbild ein %1 ist.

The syntax for formatting the first message is as follows:

// lpBuf must be large enough to hold the formatted message!
DWORD langID = MAKELANGID(LANG_GERMAN, SUBLANG_GERMAN);
HMODULE hModule = LoadLibrary(...);
TCHAR lpBuf[60];
LPVOID lppArgs[10];
DWORD len = FormatMessage(
   FORMAT_MESSAGE_FROM_HMODULE|FORMAT_MESSAGE_ARGUMENT_ARRAY,
   hModule, idMsg, langID, lpBuf, sizeof(lpBuf)/sizeof(TCHAR),
   lppArgs);

You can use FormatMessage with string tables as well as with message tables, but it is more efficient to use the function with message tables. FormatMessage can retrieve strings from message tables directly, but it cannot access string tables. To format a string from a string table, you would first have to retrieve it with the LoadString function and then pass it in a buffer to FormatMessage. Not only is this an extra step, it's a convoluted extra step.

FormatMessage is particularly useful for a number of reasons. In conjunction with the API call GetLastError, you can use it to format error messages returned by the system. An example of this technique is given in the following ReportError routine. FormatMessage also allows you to specify the language of the string you want to retrieve from a message table. LoadString can retrieve only resources associated with the language of the current thread's locale.

void ReportError()
{
LPTSTR lpMessage;
DWORD dwErrCode = GetLastError();
FormatMessage(FORMAT_MESSAGE_ALLOCATE_BUFFER |
   FORMAT_MESSAGE_FROM_SYSTEM,
   NULL,               // no source buffer needed
   dwErrCode,          // error code for this message
   NULL,               // default language ID
   (LPTSTR)&lpMessage, // allocated by fcn
   NULL,               // minimum size of buffer
   NULL);              // no inserts

MessageBox(NULL, lpMessage, TEXT("File Error"),
   MB_ICONSTOP | MB_OK );
}

.NET Framework String.Format

The .NET Framework has a counterpart to Win32's FormatMessage -the String.Format method. It allows you to number your variable placeholders so that localizers can rearrange them depending on the syntax of the language into which you are translating, without having to rearrange the variables themselves in the method call. The one difference between FormatMessage and String.Format is that instead of using FormatMessage 's percent sign (%) plus some number to designate the placeholder, String.Format encompasses the number within braces "{ }." So the earlier example would be changed from the FormatMessage syntax of:

"Not enough memory to %1 the file %2."

to the String.Format syntax of:

"Not enough memory to {0} the file {1}."

The call in your program would look like this:

String.Format("Not enough memory to {0} the file {1}", sFunc,
   sFile);

The format parameter is embedded with zero or more format specifications of the form, { N [, M ][: formatString ]}, where:

  • N is a zero-based integer indicating the argument to be formatted.
  • M is an optional integer indicating the width of the region to contain the formatted value, padded with spaces. If the sign of M is negative, the formatted value is left-justified in the region; if the sign of M is positive, the value is right-justified.
  • formatString is an optional string of formatting codes.

If the value of format is,

"Brad's dog has {0,-8:G} fleas."

arg0 is a 16-bit integer with the value 42, (and in this example, underscores represent padding spaces) then the return value will be:

"Brad's dog has 42______ fleas."

The .NET Framework has also added the idea of an argument placeholder to its Console.Write method. So an application that uses the standard input, output, and error streams for console applications can use the same syntax as the String.Format method to allow it to be more localizable, such as shown here:

Console.Write("Not enough memory to {0} the file {1}", sFunc,
   sFile);

Do Not Compound Several Variables

As stated before, there are times that variables are necessary because you might not know what needs to go in the string until run time. Things like dates, time, temperature, and number of sales are just a few examples of these types of variables. Besides giving these variables unique names, you need to make sure that you don't compound several variables together. For example, a translator ran across the following during a localization job:

"%d:%d%s on %s, %s %d, %d"

Needless to say, the translator did not know how to translate the word "on," since it was ambiguous what the word meant or referred to in this particular context. Once the programmer was contacted, the localizer learned what the variables stood for:

1st %d-Hour in 12-hour format (01-12)
2nd %d-Minute, as decimal number (00-59)
1st %s-Current locale's A.M./P.M. indicator for 12-hour clock
2nd %s-Full name for the day of the week
3rd %s-Full name for the month
3rd %d-Day of month, as decimal number (01-31)
4th %d-Year with century, as decimal number

To avoid confusion, the programmer should have used unique names. Even better, the programmer should have stored the data in language-neutral data variables-such as a time structure-and then used one of the Windows globalization services-such as National Language Support (NLS) or .NET. (For more information on these globalization services, see Chapter 4, "Locale and Cultural Awareness." ) Remember that even though you use unique variable names, you still need to leave enough information so the translators know what you are trying to say. The following string, once again, not only lacks unique names for the variables, but also compounds these variables together:

"%d %1 has %2 internal %3."

Keep Sentences in a Single String

You have seen that good localizability practices include minimizing the use of variables, providing unique names for variables, and supplying enough information for the translators to understand the context of what needs to be translated. It is also important to write things out rather than compounding variables together in a string, as in the two previous examples.

Additionally, it is important to keep sentences in a single string. When a sentence is broken up into several strings, the strings do not necessarily appear consecutively in the localizer's string table. It is very time-consuming to piece strings together to form a correct sentence. In addition, it is not possible to merely translate word by word using the same syntax as the original language, since sentence structures differ from language to language. Sentences broken up into several strings coupled with linguistic differences among languages make it hard for autotranslation tools to create one-to-one glossaries without errors.

For example, take the following English sentence that has been broken up into three strings:

"When this box is checked, Windows NT does not"
"automatically display the user name of the last person"
"to log on in the Authentication dialog box."

In order to convey the syntactical complexity and additional length in the localized version, the sentence was translated like this in German:

"Wenn dieses Kontrollkästchen aktiviert ist, zeigt"
"Windows NT nicht automatisch den Namen des"
"Benutzers an, der sich zuletzt in dem Dialogfeld"
""Authentifizierung" angemeldet hat."

Translated back into English, this sentence literally means:

"When this controlbox checked is, -plays"
"Windows NT not automatically the name of
"the user -dis, who him/herself last in the dialog box"
""Authentication" logged has."

Keeping the sentences in single strings allows automated translation tools to help your translators be more precise, more productive, and thus allow more cost savings for you.

Watch Your String Buffer Sizes

Length restrictions for strings are potential bugs that can cause major problems. For example, a product build was broken because of the German translation for the following string:

"Press Ctrl+Alt+Del to restart."

When translated into German, this string almost doubled in size to:

"Drücken Sie Strg+Alt+Entf, um den Computer neu zu starten."

The increased string size caused the buffer to overflow, which in turn crashed the system. The message was supposed to go into a boot sector of a certain byte size, and there was no space for a longer message. The only thing that can be done for this type of problem is to make sure that you communicate to localizers how much room they have for their translation.

Along these same lines, buffer overruns, when there is no physical restriction, are notorious for bugs even in nonlocalized software. Localizing strings tends to reveal buffer overrun problems. These types of problems can be eliminated if buffers are dynamic or are allowed to have the maximum buffer size. Estimating size requirements will be discussed in the next section, which will explain how an optimal UI design makes localization easier.


Localizability Considerations for the UI

Now that you have learned how to store and handle text to make localization more straightforward, you need to know how to design your UI so that the localized content does not break the look and feel of your program. The following will discuss how to minimize the resizing of your UI caused by languages that require more (or less) space for the same content; how to handle UI controls like list boxes, combo boxes, buttons and so forth; and how to effectively use images and icons.

Element Resizing

Glossary
  • Common dialog boxes: Standard dialog boxes defined by Windows for operations found in numerous applications; these operations include Open, Save As, Print, Page Setup, Color Selection, Font Selection, and Find. Applications can call common dialog box API functions directly instead of having to supply a custom dialog template and dialog procedure.

The most time-consuming part of localizing the UI is resizing elements, particularly for applications that will ship in numerous languages and that contain a large number of dialog boxes. The following provides solutions for addressing this issue when working with Win32 dialog boxes.

Win32 Dialog Boxes

You can minimize the amount of resizing necessary by creating your native-language dialog boxes with as much room to spare as you feel comfortable; extend text frames as far as possible so you can allow text to grow when it is translated. For example, in Figure 7-3, the frames surrounding the Name, Initials, and Mailing Address text fields extend all the way to the end of the edit control.

Figure 7.3 Options dialog box with extended text frames.
Figure 7.3 - Options dialog box with extended text frames.

Also, when possible, it is best to put text labels above UI controls such as edit boxes, as shown in Figure 7-3. This positioning allows for the greatest extension of the text field. Although putting text labels above UI controls is the best practice for localization, thereare times when the text label is placed to the side of a control for a better look and feel. When this is the case, make sure to leave room for the text to expand in your design, and remember to extend the text field to cover all the space given. For instance, notice the extra space allotted between the text "Digit grouping symbol" and the list-box control in Figure 7-4. In Figure 7-5, you can see much of this extra space was used when the text was translated to German.

Figure 7.4 Example of a dialog box for which extra space has been allotted between the text field and the list-box control.
Figure 7.4 - Example of a dialog box for which extra space has been allotted between the text field and the list-box control.
Figure 7.5 Example of how a text field can expand during translation.
Figure 7.5 - Example of how a text field can expand during translation.

When designing and localizing dialog boxes, you should test them with various screen sizes and resolutions to make sure they still look correct and fit on the screen, especially if your application allows the user to change the size of the application's dialog font. Microsoft Windows XP, for example, allows the user to scale the size of the system fonts, including fonts used for system-drawn items such as title bars and menus.(See Figure 7-6.)

Figure 7.6 The Advanced Appearance dialog box from the desktop Display Properties property sheet, which allows you to change the font and font size of system items such as a menu.
Figure 7.6 - The Advanced Appearance dialog box from the desktop Display Properties property sheet, which allows you to change the font and font size of system items such as a menu.

The user can also resize system dialog boxes; the dialog controls will adjust simultaneously and automatically. (See Figure 7-7.) If you decide to incorporate this functionality into your design, carefully consider how your program will automatically adjust dialog boxes that have been localized-the ease with which the dialog box designs can be made to incorporate this feature is a good indication of how localizable they are.

For each localized dialog box, double-check that the order in which the user activates dialog elements via the Tab key still makes sense. If tabbing order isn't consistent, the localizer probably changed the relative position of dialog elements to make the translations fit, or the localizer might have set the coordinates of an item outside the coordinates of the dialog box itself.

Figure 7.7 The Windows 95 Find: All Files dialog box. A user can adjust the width of the dialog box by dragging the border with the mouse.
Figure 7.7 - The Windows 95 Find: All Files dialog box. A user can adjust the width of the dialog box by dragging the border with the mouse.

To omit references to features not supported in a localized application, you could set the item's coordinates outside the dialog coordinates but, as mentioned, it is not assured that you'll have consistent tabbing order. Another way to omit references is to permanently dim such items. Although this practice preserves tabbing orders, it might confuse the user, who might not understand why certain commands are never available. As mentioned earlier, many applications have similar basic features. For example, most applications open and save files. Windows provides common dialog boxes for these operations (like the one shown in Figure 7-8).

Figure 7.8 Example of a common dialog box.
Figure 7.8 - Example of a common dialog box.

Common dialog boxes have benefits for both the user and the developer. They make applications easy to learn because they provide consistency among applications. Such dialog boxes are also easy to implement because they require only a single API function call. Common dialog boxes do, however, present a dilemma for localized applications. Currently, each edition of Windows XP ships common dialog boxes only in the default language of that edition. An application that uses common dialog boxes will, therefore, always display them in the language of the system rather than in the language in use by the application. The mix will be jarring to a user who does not understand that some dialog boxes are drawn by the application and some are drawn by the system. Another problem for applications that use common dialog boxes is that they can be embedded within application-defined resources; the system dialog boxes might differ in size on different language versions of Windows-this can cause text to be clipped in applications.

While the configuration of Windows where your application runs might not provide common dialog resources in the language of your application, one way to work around this predicament is to use templates for the common dialog boxes instead of using the common dialog boxes themselves. The application can provide translated templates to the dynamic-link library (DLL) that contains the Windows procedures for the common dialog boxes. This method might work in the short term, but it does carry the risk that the translated templates will not be compatible with future versions of the common dialog boxes.

Controls, such as radio buttons and check boxes that allow the text to wrap are another way of handling text expansion. To allow multiple lines in a Win32 resource file, you can either manually add the BS_MULTILINE property to the control (shown within bold in the following code):

CONTROL "Check1",IDC_CHECK1,"Button",BS_AUTOCHECKBOX |
   BS_MULTILINE | WS_TABSTOP,21,21,41,10
CONTROL "Radio1",IDC_RADIO1,"Button",BS_AUTORADIOBUTTON |
   BS_MULTILINE,19,45,39,10

Or in Microsoft Visual Studio you can select the Multiline property in the particular control's property sheet. (See Figure 7-9.)

Figure 7.9 Multiline property set in a Visual Studio control's property sheet.
Figure 7.9 - Multiline property set in a Visual Studio control's property sheet.

In the .NET Framework, check boxes and radio buttons are automatically set for multiple lines. Allowing multiple lines is important since localizers might not have accessto these properties; their only alternative is to leave out information so they can fit the translation onto one line. Figure 7-10 shows a dialog box that was well designed in the English-language version. When localized into German, however, the radio buttons' UI design was broken when the translated text was not allowed to wrap.

Figure 7.10 Result of not allowing controls to wrap text.
Figure 7.10 - Result of not allowing controls to wrap text.

Variables used in UI strings usually need extra space because you can never be sure how long the text will be that replaces the variable. A good rule is to add one line per variable in the text box so that there is enough space for the localized text. In Figure 7-11 the translation of the first sentence "Welcome to the %s Registration wizard." will require the entire first line. Thus the inserted text of the variable will extend the sentence onto a second line.

Figure 7.11 Example of allowing one line per variable in a text box.
Figure 7.11 - Example of allowing one line per variable in a text box.

The basic rule is that you should leave about 30 percent additional room for text expansion. Most of the time this practice works. The only exception is when you have a string or button that has less than 10 characters. Take, for example, the following OK button.

If you followed the 30-percent rule, all you would do is leave space for one more character; but this does not hold true if you are localizing the text into Spanish. The verb for "accept" in Spanish is "aceptar" (which is used within the UI to mean "OK"). However, the word "aceptar" is 250 percent bigger than the original text.

Therefore, the exception to the 30-percent rule is if the text has less than 10 characters, leave room for at least 400 percent growth. Remember that these are general rules, and all languages' UIs should be designed to look as good as possible, even in English. However, the best guideline is to leave as much space for text expansion as possible. Minimizing the need for resizing is a vital part of making sure that Web pages are localizable also.

Web Resizing Issues

Until recently, the majority of UIs on the Windows platform have been created using Win32 dialog boxes or Microsoft Visual Basic Forms. These days, many developers are taking advantage of the richness of HTML to create an application's UI. This is great for a development team, but causes huge problems for applications that need to be localized into multiple languages. That is because HTML interfaces are a lot harder to resize and readjust after translating the text.

And it's not just UIs that are hard to resize after translation. Ordinary Web pages that utilize the full power of cascading style sheets (CSS) can be an absolute nightmare! People have found that the cost of translating, resizing, and testing HTML UIs is far greater than that of Win32 dialog boxes.

Nevertheless, with the right coding standards, all HTML UI elements can be constructed so that the localization team will have little or no resizing and repositioning work to do. One approach that Microsoft uses internally to accomplish this is called "HTML AutoLayout (HAL)." The HAL documentation shows some of the key coding standards you should follow to allow for "zero resizing." (To view the HAL documentation, see the Extras subdirectory on the companion CD, or go to http://www.microsoft.com/technet.)

The HAL documentation lists a series of rules and the key reasons behind them. These rules are:

Rule 1: Never use absolute positions.

Rule 2: Dialog boxes should use the available width and height.

Rule 3: Each control should be in a separate cell.

Rule 4: Allow wrapping text.

Rule 5: Separate check boxes and radio buttons from labels.

Rule 6: Leave room for growth and avoid fixed-width items.

Rule 7: Define methods for sizing buttons, list boxes, and group boxes.

Rule 8: Define when to set height.

Rule 9: Do not use left or right align exclusively.

Rule 10: Avoid inline CSS with values that localization needs to change.

The rules are then followed by a set of coding samples.

A key point to remember is that HAL is just one approach for achieving zero international resizing. There are probably countless other methods. This is fine as long as they achieve the same result. Even so, other methods will probably benefit from using some of the HAL rules. Another point to bear in mind is that there are already many HTML dialog boxes that are HAL-compliant for the most part. This is especially true for dialog boxes (or windows) that were designed to be resizable.

Localizability of UI Controls

In addition to element resizing, another major localizability consideration for the UI involves UI controls. Since programmers do not have to create their own controls, such as list views and edit boxes, their job is made somewhat easier. However, when used in certain ways, these same controls can pose a problem for localizers. Edit boxes, for instance, often need to be repositioned dependingon the syntax of the target language. The following sections will give you some tips on what to be aware of when working with UI controls.

UI Controls Within Sentences

The biggest problem with UI controls in terms of localizability is when programmers use them as part of a sentence. Localizers often have to either change the position of the controls (if they have access to a formatting tool) or settle for an improper sentence structure. For example, Figure 7-12 is designed with an edit control in the middle of a sentence. In its German localization, the edit control had to be repositioned so that what it represented could fit syntactically in the German translation. (See Figure 7-13.)

Figure 7.12 This edit control occurs in the middle of a sentence.
Figure 7.12 - This edit control occurs in the middle of a sentence.
Figure 7.13 When localized into German, the edit control had to be repositioned.
Figure 7.13 - When localized into German, the edit control had to be repositioned.

A better design is to rethink the whole phrase, so that the edit box can be removed from the sentence. (See Figure 7-14.) This way the localizer does not have to reposition the edit box. (See Figure 7-15.)

Figure 7.14 The edit control appears outside the sentence.
Figure 7.14 - The edit control appears outside the sentence.
Figure 7.15 When localized into German, there is no need to reposition the edit control.
Figure 7.15 - When localized into German, there is no need to reposition the edit control.

The other problem with using UI controls in sentences is when you have a drop-down box. As mentioned earlier, strings built at run time cause problems because of grammatical differences (such as the existence of gender in some languages) and syntactic differences (such as verbs that can either precede or follow a noun depending on the language and context). Similarly, drop-down boxes might have terms in them that could change the translation of the sentence depending on the term selected. Plus, moving andaligning drop-down combo boxes that usually consist of multiple controls is always error-prone.

Hidden or Overlapped UI Controls

UI controls such as buttons or drop-down lists should not be placed on top of other controls. Some localization tools are not able to change the state of a dialog box; thus the localizers might not even know that there are hidden controls to translate. Even when localizers have access to all the controls, sizing and hot-key issues with hidden controls are usually only found through testing. In Figure 7-16, the UI is not localizable because the button size cannot be extended to the length required for the translation without rearranging the button positions; also, rearranging is not possible because there is not enough space in the dialog box. Although rearranging button positions is costly and makes the UI inconsistent among languages, there might be times when it is necessary to do so.

Figure 7.16 This dialog box is not localizable because the button size cannot be extended without rearranging the position of the buttons, and there is not enough space for rearrangement.
Figure 7.16 - This dialog box is not localizable because the button size cannot be extended without rearranging the position of the buttons, and there is not enough space for rearrangement.

Figure 7-17 demonstrates the other problem with hidden controls. Even if there were room for the expansion of the Edit Properties button, unless the Accept button is resized also, the Edit Properties button would be partially visible when it actually should be hidden. This causes a very unusual UI in which buttons overlap.

Figure 7.17 Dialog box in which the Edit Properties button and the Accept button overlap.
Figure 7.17 - Dialog box in which the Edit Properties button and the Accept button overlap.

Button Text

Avoid placing button text into a string variable. Text on a button should never be dynamically linked onto the button from a string variable, but should be placed on the button itself as a property of the button. Button sizes usually have to be adjusted to fit the length of the translation on the button. Localizers have no way of telling which strings in the string table end up on which button at run time. (See Figure 7-18.)

Figure 7.18 The string
Figure 7.18 - The string "add job" ("Auftrag hinzufügen") was stored as a text resource. Since the button cannot change its size at run time, part of the text is truncated.

Images and Icons

Since the beginning of graphical user interfaces (GUIs), images and icons have been a major part of well-designed architecture. Their greatest strength is that with one picture a whole concept can be depicted. One reason behind this is that a picture evokes shared experiences that everyone has within their particular culture. When it comes to software localizability, this strength from shared experiences becomes one of the greatest weaknesses of images and icons. Obviously not all cultures share the same experiences or backgrounds. What for one culture might be a positive experience might be a negative experience for another.

Take colors, for example. In the United States, wedding attire for the bride is largely associated with the color white, while black is associated with death and burials. However, in Japan the color symbolism is just the opposite; white is the predominant color at funerals, and black is considered a color that brings good luck for weddings. Cultural color perception is just one of many issues that can affect the perception and effectiveness of your overall design. The following sections will look at issues to avoid when designing images and icons for your interface.

Avoid Culture-Specific Examples

Just as color can evoke different emotions from culture to culture, certain images can have the same effect. A couple of good examples come from the early days of GUI operating systems. In earlier UIs a U.S. rural mail box was used to indicate mail. This was acceptable for use in the United States because many people had grown up using these types of mail boxes. However, when the image was introduced in Europe, most people wanted to know what a breadbox sitting on a pole had to do with mail. A much better choice would have been-and still is-using an outline of a postal letter, since most people who understand the concept of mail would understand this image. (See Figure 7-19).

Figure 7.19 Earlier image used to represent mail, versus the newer and more culturally-appropriate image.
Figure 7.19 - Earlier image used to represent mail, versus the newer and more culturally-appropriate image.

Another example of an image that can have varying interpretations from one culture to another involves the concept of a "wizard." A wizard was used in earlier UIs to represent assistance for doing something the user might not understand. Based on Americans' experience with wizards like Merlin in the King Arthur stories, pointed caps and magic wands were used to depict this concept. (See Figure 7-20.) Again, these images worked well here in the United States, but the concept of a wizard is foreign to many other cultures. Even for some cultures that do understand the concept, wizards don't necessarilywear pointy hats. In still other cultures, a wizard represents something evil or destructive; it can even be a religious abomination, rather than something that is meant to enlighten or assist someone.

Figure 7.20 Culture-specific images formerly used to depict a software wizard.
Figure 7.20 - Culture-specific images formerly used to depict a software wizard.

Avoid Showing Flesh or Body Parts

There are as many different attitudes about displaying the exposed human body as there are shades of gray. What is considered taboo in one culture or country can be widely accepted in another. These varying attitudes range from the body needing to be completely covered to the acceptance of bikinis. With the widespread use of the Internet and global communications, you will never know for sure who will see or use your products. Thus the best practice is not to display skin or even use a body part to convey a concept. So instead of using an open hand to represent "stop," use the internationally-accepted, eight-sided stop sign. (See Figure 7-21.) In addition to the taboo factor, although using the image of an open hand might be understood in the United States as a way to indicate the action of stopping, this doesn't mean the image will be understood in other cultures.

Figure 7.21 Rather than using the hand to represent the action of stopping, use the internationally-accepted stop sign.
Figure 7.21 - Rather than using the hand to represent the action of stopping, use the internationally-accepted stop sign.

The use of body parts can indeed raise different emotions other than the one intended. Take the American gesture for "okay." (See Figure 7-22.) Although the gesture might be accepted in the United States, there are other parts of the world where it will not be understood, or where it can even have some very obscene connotations. Again, it is better to avoid using images involving a human body part, unless you are dealing with software that has a specific medical theme or context. Even then, be careful not to show a hand in some type of gesture.

Figure 7.22 Gesture used in the United States for
Figure 7.22 - Gesture used in the United States for "okay."

Beware of Gender-Specific Roles and Ethnic Stereotypes in Other Cultures

Taking the skin and body gestures one step further, you must also be aware that diverse cultures view the roles of men and women quite differently. In some cultures, men and women are treated as complete equals; in others, one gender is seen as subservient to the other. Thus in some cultures Figure 7-23 wouldbe offensive because it shows a woman standing above men in an apparently superior role. A better graphic to portray the same idea is in Figure 7-24, where the human figures are gender-neutral.

Figure 7.23 Example of a potentially offensive graphic for its depiction of gender-assigned roles.
Figure 7.23 - Example of a potentially offensive graphic for its depiction of gender-assigned roles.

Figure 7-24 raises another point: ethnicity. In this figure you cannot tell what race the people belong to. Some cultures feel the same way about ethnic-group roles in society as they do about gender roles. By adhering to the guideline of not showing skin (or skin color), you can feel safer that you are not generating content that has higher potential risk in a locale.

Figure 7.24 Example of an inoffensive graphic, since it avoids depicting gender-assigned roles.
Figure 7.24 - Example of an inoffensive graphic, since it avoids depicting gender-assigned roles.

Avoid Religious References

From ethnicity, the content risk can be widened to include religious references. It's important to avoid any religious references so that you don't offend a local culture's expectations or appease one religious group at the expense of another. A symbol that might seem innocent to a developer in the United States might be sacrilegious to a consumer in a certain market. Be very cautious about employing any religious references, even for holidays and more common events.

Avoid Political Symbols

When it comes to governments and politics, it is very important to realize that your product's distribution in a local market is often dependent upon the local government's approval. Thus when localizing for a specific market, you must take into account the local government's expectations for content. As much as possible, it's important to avoid including content that could offend the local government-either by challenging its authority or by criticizing it through support of a rival government or faction.

Software has been banned in some countries and regions simply because a map showed that a disputed piece of land belonged to another country. Maps are very graphic and obvious statements about a government's sovereignty, so a user associated with the disputed piece of land would know very quickly if the maps are accurate or not. In addition to maps, flags can be a very sensitive piece of content. For example, a flag in a UI that represented an unrecognized country was very upsetting to a nearby government, causing that government to ban a product on the basis of the unrecognized national flag. Both of the incidents just mentioned emphasize one of the most misused group of graphics: political symbols, maps, and flags. Most unknowing developers use flags as a graphical way to display a particular language, country, or region, but that is not what a flag represents. Flags are nationalistic; they represent ideals, boundaries, and political beliefs, but they do not represent a language. For instance, which flag do you use to represent the English language-the American flag, the British flag, the Canadian flag, or the Australian flag? All of these countries speak English. In the process of selecting the most appropriate flag, you will inevitably offend someone because you left them out. The best practice with all flags, national symbols, maps, and so forth is to avoid them as much as possible.

Avoid Text in Graphics

The last area to consider when dealing with graphics involves embedded text. Figure 7-25 displays an icon that a developer wants to use to indicate the function of opening a folder. To make sure that the user understands what to do, the developer has added the word "open" to the graphic.

Figure 7.25 Translating this embedded text will cost you valuable time and resources.
Figure 7.25 - Translating this embedded text will cost you valuable time and resources.

Using embedded text such as this is helpful if you are only dealing with the English language. However, when you localize your software for another language, you have to translate this graphic for each language edition. Depending on how the graphic was created, this could take some time. Certain graphics packages-such as Microsoft Paint-often take the text you add and convert it to individual pixels. This means that localizers will need to go back into the graphic, change all the pixels from the text color to the background color, and then add the new translation. Furthermore, they must do this for each language into which you are translating.

Suppose you are translating into 24 languages and it takes 10 minutes to adapt each graphic. That one graphic now costs you 240 minutes or four hours of resources. What seemed to be a simple translation has become a half-day project. If you had to adapt just 10 graphics, you have now unnecessarily added five days of work. In addition, even more time might be involved if the background is more complicated, such as if it has many colors, rather than being monochromatic. Besides the extra time spent localizing the graphic, another issue is that there are now 24 different graphics that you have to keep track of. If you have 10 graphics, you would need to deal with 240 different graphics to make sure that they are in the right place and that they are part of the right build.

The better way to handle this is not to add text to a graphic at all. For the previous example, that would save you $2000-using a $25-an-hour estimate for labor-to localize 10 graphics for 24 languages. It would also save you the hassle of keeping track of those 240 different graphics. Even if you think there is a verygood reason to have text in a graphic, you should ask yourself if it is absolutely essential. If so, you can still minimize the extra work needed. For instance, some graphics packages have a function that stores the text as a separate layer from the graphic itself. This way, when you need to go back into the graphic, you don't have to change it pixel by pixel. You only have to go into the text layer and change it to the new translation. Although storing the text as a separate layer saves you some time in editing, you still must do this for each language you are localizing into, and for each graphic for which you have text. This practice does not spare you from having to track all additional graphics.


Summary

Though localization presents some inevitable challenges, by following good software localizability practices from the beginning stages of software development, you can avert or scale such obstacles. These practices can improve the quality of your translation and dramatically decrease localization costs.

Although localizability bugs aren't found until late in the development process, they are generally easy to fix. However, one way to prevent localizability bugs is to separate all localizable items into one or more files. The simplest method of dealing with localizable resources is to put everything in resource repositories, which can be separated into four categories: UI resources, resources used for the adaptation of a product, debug resources, and functional resources. The goal is to separate the first two categories from the last two categories and to make sure that resources clearly belong to one category only. Other recommendations are to put resources that do not need localization into header files and to leave resource identifiers unchanged.

When dealing with strings, avoid run-time composite strings. When variables are necessary, use unique names. Do not compound several variables. Keep sentences in a single string, and watch your string buffer sizes. In terms of UI localizability, minimize the amount of resizing necessary by creating your native-language dialog boxes with as much room to spare as you feel comfortable. Extend text frames as far as possible to allow text to grow when it is translated. The basic rule for accomodating text extension caused by translation is to allocate about 30 percent additional room. To prevent mistranslation and the need for repositioning of UI controls, avoid using UI controls as part of a sentence. UI controls such as buttons or drop-down lists should not be placed on top of other controls. Also, do not place button text into a string variable.

Finally, when you include icons or images, avoid culture-specific examples, and the display of flesh or human body parts. Beware of gender-specific rulesand ethnic stereotypes in other cultures. Avoid religious references, political symbols, and text in graphics. By keeping all these guidelines in mind as you design and write software, you can be proactive in making sure your software is localizable.


Chapter 8: Mirroring

Books that are published in languages that use a Latin script such as English, Italian, or Dutch have certain conventions that differ from books published in such languages as Arabic, Hebrew, Farsi, or Urdu. While the first category involves left-to-right (LTR) languages, the second category involves right-to-left (RTL) languages; consequently, the text will need to be laid out accordingly. Additionally, page numbers within books written in English, Italian, and Dutch increase as you progress from left to right within the book, while the opposite holds true for Arabic, Hebrew, Farsi, and Urdu.

Within software applications, the user interface (UI) for RTL languages will differ from the UI for LTR languages. If you plan to localize into an RTL language, one very essential technique in making software localizable involves UI mirroring. This technology only applies to RTL languages including Arabic, Hebrew, Farsi, and Urdu; it isn't used for LTR languages. If you are not planning any localization into RTL languages, you can safely ignore mirroring-specific requirements.

To give a perfect RTL look and feel to an application's UI, both the text and the UI elements need to be laid out from right to left once they are translated into RTL languages. The mirroring technology allows you to provide such an interface to your Microsoft Win32 application, to your Web-based application, and within the .NET Framework as well.

This chapter will first look at the technology of mirroring and its overall implementation in Microsoft Windows. It will then focus on factors you will need to consider and coding practices you should use to provide a mirrored application.


The Technology

When the Arabic and Hebrew versions of Microsoft Windows 95 were localized, a mechanism known as "flipping" was used to provide the UI with a right-to-left look and feel. With flipping, items in a dialog box were repositioned within a dialog template so that UI elements would have a natural flow. However, this partial solution did not address layout problems for such items as title bars, tree views, or combo boxes. It merely repositioned these items within the flipped dialog box. To resolve the remaining issues that flipping was unable to address, the mirroring technology was introduced in the Arabic and Hebrew versions of Microsoft Windows 98. However, since Microsoft Windows 2000 and Microsoft Windows XP provide underlying support for layout, and since both operating systems are single binaries, mirroring is now available on all language versions of Windows 2000 and later. It is activated at the application level.

Figure 8-1 displays an Arabic Windows XP desktop where every UI element-the Start button, tree-view controls, title bar, scroll bars, and toolbars-is mirrored.

Figure 8.1 An Arabic Windows XP desktop in which each UI element is mirrored.
Figure 8.1 - An Arabic Windows XP desktop in which each UI element is mirrored.

The code modifications required to mirror UI windows, while not trivial, were less extensive than you might think. The Bidirectional (BiDi) Windows 98 development team first designed an architecture and programming interface to make most of the layout more or less automatic. The basis of this architecture is a simple coordinate transformation: the origin of a window changes from the upper-left corner to the upper-right corner, and the x scale factor changes such that x values increase from right to left, rather than from left to right. (See Figure 8-2.) Also, by mirroring a window, you can affect how text is rendered within that window. A mirrored window gets a caption that is rendered in RTL direction; the same applies to labels, button captions, and text on menus. Win32 application programming interfaces (APIs) hide implementation details, transparently allowing applications to take advantage of the same mirroring functionality that the system uses.

Figure 8.2 The architecture for mirroring is based on a coordinate transformation.
Figure 8.2 - The architecture for mirroring is based on a coordinate transformation.

When mirroring Win32 applications, you can either choose to enable mirroring in your resources or in your code. The following sections describe both of these methods and show the circumstances under which you would generally use each method. You will also see how to mirror property sheets, what to consider when mapping coordinates, how to handle direction-sensitive graphics, and how to work with image-list controls.


Mirroring in Win32 Applications

To minimize the amount of rewriting needed for applications to support mirroring, system components such as the GDI and User modules have been adapted to enable control over the mirroring state of the UI objects. Almost no additional code changes are necessary, with the exception of a few things you will need to consider when working with owner-drawn controls and bitmaps. As you will see, new APIs and extended styles for windows let you turn mirroring on and off at the process level, the window level, the dialog-resources level, or even the device-context level. However, before examining the capabilities of system components and APIs that are available when you enable mirroring in code, this chapter will take a quick look at how and when to enable mirroring in resources.

Enabling Mirroring in Resources

With changes that the Windows team has made to the system's resource loader, you can mirror your applications by simply editing your resources. This is a good approach to use when you want to mirror old components for which the source is not available or is no longer used; it is also beneficial when you want to conduct primary testing of your applications to see if they support mirroring.

You can mirror applications by adding two left-to-right marks (LRMs)-represented by Unicode code point U+200E-in front of the FileDescription value of the version stamping. Figure 8-3 shows an English Notepad.exe file that has been mirrored with this approach.

Figure 8.3 A Notepad.exe file that has been mirrored by adding two LRMs.
Figure 8.3 - A Notepad.exe file that has been mirrored by adding two LRMs.

Although easy, enabling mirroring in resources offers little flexibility when it comes to just mirroring a specific window or control. In fact, all windows and controls within those windows that belong to the affected process will be automatically mirrored. For that reason, enabling mirroring in code is more flexible than mirroring in resources, since it allows you the freedom to decide what you want mirrored. Keep in mind that a child process created by the mirrored process does not inherit the mirroring style.

Important
Ifyour sole purpose is to test the behavior of your UI in a mirrored environment, mirroring processes by making changes to resources is a quick way to implement mirroring. However, this is merely a workaround and is not a Microsoft-supported development technique. The proper way to implement mirroring would be by enabling your code, as explained in the next section.

Enabling Mirroring in Code

Glossary
  • Noncompile mirroring: Activating mirroring for the binary dynamic-link library (DLL) resource files of an application, without the need for rebuilding the application.
  • Device context: A Graphics Device Interface (GDI) structure that defines a set of graphics objects and their associated attributes, as well as the graphics modes that affect output. The graphics objects include a pen for line drawing, a brush for painting and filling, a bitmap for copying or scrolling parts of the screen, a palette for defining the set of available colors, a region for clipping and other operations, and a path for painting and drawing operations.
  • MM_TEXT: Unit of measure used to convert logical units to device units; it also defines the orientation of the device's x - and y -axes. GDI uses the mapping mode to convert logical coordinates into the appropriate device coordinates. The MM_TEXT mode allows applications to work in device pixels, where 1 unit is equal to 1 pixel. The physical size of a pixel varies from device to device.

Although you can activate mirroring through the localization process (with non- compile mirroring), doing so will often reveal some issues that require a code fix. Also, as stated, you might want full control over when to enable mirroring-such as per process, per window, for dialog resources, or per device context, as shown in the following sections-and over which elements you want mirrored. This is only possible through coding and through new mirroring APIs.

Activating Mirroring per Process

By activating the mirroring layout on a per-process basis, all windows that are created after activation will be mirrored; but this activation will not affect the existing windows. This approach is very similar to noncompile mirroring, with the exception that the mirroring style within an application can be turned on and off at run time rather than at a binary level, as in the case of noncompile mirroring.

You set the default direction to RTL for a process by calling SetProcessDefaultLayout(LAYOUT_RTL). You can also turn off default mirroring by calling SetProcessDefaultLayout(0). In addition, you can query the current default layout direction as follows:

BOOL WINAPI GetProcessDefaultLayout(DWORD *pdwDefaultLayout);

After returning successfully, pdwDefaultLayout contains either LAYOUT_RTL or 0.

Although you can activate mirroring at the process level, there are times you might wish to activate it at a lower level-such as per window. The following section explains when this method is a good option and shows you what extended styles to use.

Activating Mirroring per Window

Once the mirroring style is set at a process level, all newly created windows are automatically mirrored. An alternative approach is to activate mirroring on a per-window basis (if not all windows in your application are meant to be mirrored). With this approach, a window can be mirrored upon its creation by calling CreateWindowEx, and set by using the extended style WS_EX_LAY- OUTRTL.

By default, all child windows of a parent window inherit the mirroring attribute. However, once again, not all windows are necessarily meant to be mirrored; thus you might want to prevent child windows from being mirrored, for example. To remove the mirroring inheritance for child windows, you need to specify the new extended style WS_EX_NOINHERITLAYOUT in your call to CreateWindowEx.

A good example of a case where you should prevent the mirroring of child windows would be when you have windows that host or import external controls, such as Microsoft ActiveX, Java applets, Component Object Model (COM) objects, and so on. The problem with controls such as these is that you have little ability to manipulate their behavior, and you cannot readily impose mirroring requirements on them.

The following code shows how to toggle the layout of a window between RTL and LTR:

LONG lExStyles;

// Retrieve the extended style of the window.
lExStyles = GetWindowLong(hWnd, GWL_EXSTYLE);
// Toggle layout.
lExStyles ^= WS_EX_LAYOUTRTL;
// Set extended styles to new value.
SetWindowLong(hWnd, GWL_EXSTYLE, lExStyles);
// Update client area.
InvalidateRect(hWnd, NULL, TRUE);
Important
Changing the mirroring style on the fly is not always the most efficient way of implementing mirroring of windows. Many controls might not behave as expected and could be improperly mirrored. Therefore, enabling mirroring of windows as they are initially created remains the better approach.

Stepping down yet another level, you can activate mirroring at the level of a window's dialog resources. The following section shows you how to do this and identifies which extended styles to use.

Activating Mirroring for Dialog Resources

It's important to distinguish between a window and a dialog box (which does not receive WM_CREATE, but rather WM_INITDIALOG). To activate mirroring for dialog boxes, you need to set the mirroring flags in your resource editor and add the extended styles WS_EX_LAYOUTRTL and WS_EX_NOINHERITLAYOUT. (The second style is only if you want to prevent the inheritance from taking place.)

The only way to switch between mirrored and nonmirrored dialog resources at run time is to have two sets of dialog resources: one mirrored and one nonmirrored. The Dialog Properties property sheet within the resource editor of Microsoft Visual Studio 6 allows you to set the mirroring style of the dialog resources. (See Figure 8-4.)

Figure 8.4 Setting the mirroring style of the dialog resources within the resource editor of Visual Studio 6.
Figure 8.4 - Setting the mirroring style of the dialog resources within the resource editor of Visual Studio 6.

Another level at which you can enable mirroring is at the level of the device context. The following pinpoints when and how to do this.

Activating Mirroring per Device Context

The mirroring technology described up to now applies to all objects in a standard window. These objects, as well as bitmaps and icons in a mirrored window, are all mirrored by default. Obviously, there are some graphics objects, particularly those that include text, that shouldn't be mirrored. Therefore, Microsoft provides several GDI functions in platforms that support mirroring (which, again, includes all versions of Windows 2000 and Windows XP). These functions are shown in the following code:

 BOOL WINAPI SetLayout(HDC hDc, DWORD dwLayout);
 DWORD WINAPI GetLayout(HDC hDc);

The device context of a mirrored window is mirrored by default. Nevertheless, you can disable mirroring in a device context by calling SetLayout (hdc, 0), and activate it by calling SetLayout (hdc, LAYOUT_RTL). You might need control over the mirroring properties of a device context in cases when some drawing operation performed by a legacy code is broken by mirroring.

Drawing images that may be sensitive to directionality of the output presents a separate problem. To prevent mirroring of bitmaps, call SetLayout with the LAYOUT_BITMAPORIENTATIONPRESERVED bit set in dwLayout :

 SetLayout(hdc, LAYOUT_RTL | LAYOUT_BITMAPORIENTATIONPRESERVED);

Upon successful completion of GetLayout, GetLayout returns the following:

  • 0 if the device context is not mirrored
  • LAYOUT_RTL if the device context is mirrored
  • LAYOUT_RTL and LAYOUT_BITMAP if the device context is mirrored and if the programmer does not want the device context to have mirrored bitmaps

Yet another way to prevent a bitmap from being mirrored is to define NOMIR-RORBITMAP in calls to BitBlt and StretchBlt.

Now that you know how and when to activate mirroring for processes, windows, dialog resources, and device contexts, what are some other factors to consider when mirroring? For instance, how can you use mirroring for property sheets?

Mirroring and Property Sheets

A tricky situation when enabling mirroring is when you are dealing with property sheets (and wizards). Imagine the case where a property-sheet container has several pages, and only a few of them are localized into an RTL language. Individual pages are made out of dialog templates that can be mirrored separately in the resources. But what about the container? Should the container be mirrored? To avoid confusion and inconsistent behavior with property sheets, the following approaches are among many steps that the system takes in order to decide whether to create a mirrored container:

  • If the first page does not contain the mirroring window style and the client process is not mirrored, the client will not be considered a mirrored client. This can happen when the application loads the first page from a different binary.
  • If the primary resource language of the first page is not Arabic or Hebrew, the client will not be considered a mirrored client. This can happen when the application loads the first page from a different binary (a binary which might not be localized for the same language).

The GlobalDev sample application shows how to create a property-sheet client that is mirrored. (To view this application, see the Samples subdirectory on the companion CD.) Figure 8-5 illustrates a GlobalDev application created in a fully mirrored container when the UI language is set to Hebrew.

Figure 8.5 GlobalDev application created in a fully mirrored container when the UI language is set to Hebrew.
Figure 8.5 - GlobalDev application created in a fully mirrored container when the UI language is set to Hebrew.

Another area to consider when mirroring Win32 applications involves mapping coordinates. The following shows you how to handle what can sometimes be another tricky matter.

Mapping Coordinates

Glossary
  • Screen coordinates: The system and applications specify the position of a window on the screen in screen coordinates. The full position of a window is often described by a RECT structure containing the screen coordinates of two points that define the upper-left and lower-right corners of the window.
  • Client coordinates: Relative coordinates of a window or client area as specified by the system or applications. Client coordinates ensure that an application can use consistent coordinate values while drawing in the window, regardless of the position of the window on the screen.

Perhaps one of the biggest issues when it comes to dealing with mirroring is mapping between screen coordinates and client coordinates, or failing to do so. For example, developers often use code such as the following to position a control in a window:

 GetWindowRect(hControl, (LPRECT) &rControlRect);
 // Gets coordinates of window in screen coordinates.
 ScreenToClient(hDialog, (LPPOINT) &rControlRect.left);
 // Maps screen coordinates to client coordinates
 //    in dialog box.
 ScreenToClient(hDialog, (LPPOINT) &rControlRect.right);

In this code, the left edge of a rectangle becomes the right edge in a mirrored window, and vice versa, causing unintended results. The solution is to replace the ScreenToClient calls as shown in the following code:

 GetWindowRect(hControl, (LPRECT) &rControlRect);
 // Gets coordinates of window in screen coordinates.
 MapWindowPoints(NULL, hDialog, (LPPOINT) &rControlRect, 2);

This code works because the MapWindowPoints API has been modified on platforms that support mirroring. As a result of this modification, Map-WindowPoints is able to swap the left- and right-point coordinates of the client window that is being mirrored, whenever two points at a time are passed to this API.

Another common practice that can cause problems in mirrored windows is using offsets in screen coordinates to position objects in a client window, instead of using client coordinates. For example, to position a control in a dialog box, the following code uses the difference in the left edges of the control and the dialog box-in screen coordinates-as the x position in client coordinates:

 RECT rcDialog;
 RECT rcControl;
 HWND hControl = GetDlgItem(hDlg, IDD_CONTROL);
 GetWindowRect(hDlg, &rcDialog);
 // Gets rect in screen coordinates
 GetWindowRect(hControl, &rcControl);
 // Takes x
 position in client coordinates
 MoveWindow(hControl, rcControl.left - rcDialog.left,
 rcControl.top - rcDialog.top, nWidth, nHeight, FALSE);

This works fine when the dialog window has an LTR layout, and when the mapping mode of the client is MM_TEXT. The new x position in client coordinates corresponds to the difference in the left edges of the control and the dialog box in screen coordinates. In a mirrored dialog box, however, the roles of left and right are reversed. You can remove the assumption from the previous code that near is left and far is right by using MapWindowPoints to go into client coordinates, as shown in the following code sample:

 RECT rcDialog;
 RECT rcControl;
 HWND hControl = GetDlgItem(hDlg, IDD_CONTROL);
 GetWindowRect(hControl, &rcControl);
 // MapWindowPoints will work correctly in mirrored
 // windows and in non-mirrored windows.
 MapWindowPoints(NULL, hDlg, (LPPOINT)&rcControl, 2);
 //Now rcControl is in client coordinates.
 MoveWindow(hControl, rcControl.left, rcControl.top,
    nWidth, nHeight, FALSE);

There is a common theme in these examples. Instead of thinking in terms of the concrete notions of left and right, you should substitute the more abstract concepts of near and far. The following are some specific guidelines on developing applications that can be mirrored by using the automatic mirroring interfaces:

  • Use client coordinates rather than screen coordinates wherever possible.
  • If necessary, map from screen coordinates to client coordinates.
  • To map points, use MapWindowPoints instead of ClientToScreen and ScreenToClient.

Another area to consider when mirroring is how to deal with direction-sensitive graphics. Suppose you have an application that contains a relatively large number of graphics. How these graphics are mirrored (or whether they are at all) will vary according to circumstances.

Handling Direction-Sensitive Graphics

Direction-sensitive graphics (such as bitmaps and icons) present another challenge with regard to mirroring. Some direction-sensitive graphics can have a different meaning when mirrored. For example, within an LTR layout in a browser, an arrow that points to the left represents the concept of going back to the previous page; an arrow that points to the right would signify going forward to the next page. When these arrows are mirrored for an RTL layout, the meaning will be just the opposite. (See Figure 8-6.)

Figure 8.6 Example of a direction-sensitive arrow that changes meaning when mirrored.
Figure 8.6 - Example of a direction-sensitive arrow that changes meaning when mirrored.

Other graphics should not be mirrored at all, such as legal trademarks or logos because they will make no sense when laid out from right to left. (See Figure 8-7.)

Figure 8.7 Example of a direction-sensitive graphic that should not be mirrored.
Figure 8.7 - Example of a direction-sensitive graphic that should not be mirrored.

There are several solutions available for dealing with cases such as these. Deciding which of the following four approaches to use depends on the particular situation at hand.

One solution is to temporarily turn off RTL layout of the device context when rendering direction-sensitive bitmaps or icons; you can then turn it back on once mirroring is completed. The problem with this approach is that sometimes you don't have access to the device context where the graphics will be rendered. (An example of this is when passing your icon or bitmap handle to a third-party DLL that will display the image for you.)

Another solution is to have two different sets of graphics in your resources-one set to be used when the drawing destination is LTR-oriented, and the other set to be used when the drawing destination is RTL-oriented. The first set will contain the original, LTR graphics. The second set will contain a copy of the first set of graphics with modifications made only to those graphics needing to reflect the RTL orientation. You can then choose which set to load and render based on whether the set will be rendered in a mirrored or nonmirrored situation. In other words, for nonmirrored device contexts you would load the original set of nonmirrored graphics, and for mirrored device contexts you would load the set that contained the assortment of mirrored and nonmirrored graphics. (In the latter case, when mirrored graphics are rendered on a mirrored device context, they will be mirrored again and thus revert back to the way you want them to be displayed.) Depending on the number of direction-sensitive graphics that you have in your project, this approach might not be your best choice if you have a lot of graphics that should not be mirrored, since duplicating those will increase the size of your resources.

As a third solution, if you ship a separate set of binaries based on the target localized languages, you can let your localization team mirror those graphics elements using a resource editor. (Most resource editors let you flip images in both x and y directions nowadays.) A final solution-especially when you don't have direct access to the device context where the graphics will be rendered-is to create a mirrored copy of the resource. You then pass this mirrored copy instead of the original, nonmirrored one. (Again, when the mirrored graphics are rendered on a mirrored device context, they will revert back to their original, intended look.) The following sample code illustrates how to mirror both a bitmap and an icon when each's handle is passed:

HBITMAP CreateMirroredBitmap( HBITMAP hbmOrig)
{
   HDC     hdc, hdcMem1, hdcMem2;
   HBITMAP hbm = NULL, hOld_bm1, hOld_bm2;
   BITMAP  bm;

   if (!hbmOrig)
      return NULL;
   if (!GetObject(hbmOrig, sizeof(BITMAP), &bm))
      return NULL;

   // Grab the screen DC.
   hdc = GetDC(NULL);

   if (hdc)
   {
      hdcMem1 = CreateCompatibleDC(hdc);

      if (!hdcMem1)
      {
         ReleaseDC(NULL, hdc);
         return NULL;
      }

      hdcMem2 = CreateCompatibleDC(hdc);
      if (!hdcMem2)
      {
         DeleteDC(hdcMem1);
         ReleaseDC(NULL, hdc);
         return NULL;
      }

      hbm = CreateCompatibleBitmap(hdc, bm.bmWidth, bm.bmHeight);

      if (!hbm)
      {
         ReleaseDC(NULL, hdc);
         DeleteDC(hdcMem1);
         DeleteDC(hdcMem2);
         return NULL;
      }

      /*
       * Flip the bitmap.
      */
      hOld_bm1 = (HBITMAP)SelectObject(hdcMem1, hbmOrig);
      hOld_bm2 = (HBITMAP)SelectObject(hdcMem2 , hbm );

      SetLayout(hdcMem2, LAYOUT_RTL);
      BitBlt(hdcMem2, 0, 0, bm.bmWidth, bm.bmHeight, hdcMem1,
         0, 0, SRCCOPY);
      SelectObject(hdcMem1, hOld_bm1 );
      SelectObject(hdcMem1, hOld_bm2 );
      DeleteDC(hdcMem1);
      DeleteDC(hdcMem2);

   ReleaseDC(NULL, hdc);
   }
   return hbm;
}

HICON CreateMirroredIcon(HICON hiconOrg)
{
   HDC      hdcScreen, hdcBitmap, hdcMask = NULL;
   HBITMAP  hbm, hbmMask, hbmOld,hbmOldMask;
   BITMAP   bm;
   ICONINFO ii;
   HICON    hicon = NULL;

   hdcBitmap = CreateCompatibleDC(NULL);
   if (hdcBitmap)
   {
      hdcMask = CreateCompatibleDC(NULL);

      if( hdcMask )
      {

         SetLayout(hdcBitmap, LAYOUT_RTL);
         SetLayout(hdcMask, LAYOUT_RTL);
      }
      else
      {
         DeleteDC( hdcBitmap );
         hdcBitmap = NULL;
      }
   }

   hdcScreen = GetDC(NULL);

   if (hdcScreen)
   {
      if (hdcBitmap && hdcMask)
      {
         if (hiconOrg)
         {
         if (GetIconInfo(hiconOrg, &ii) &&
               GetObject(ii.hbmColor, sizeof(BITMAP), &bm)
         {
               /*
                * Do the cleanup for the bitmaps.
               */

               DeleteObject( ii.hbmMask );
               DeleteObject( ii.hbmColor );
               ii.hbmMask = ii.hbmColor = NULL;

               hbm = CreateCompatibleBitmap(hdcScreen,
                        bm.bmWidth, bm.bmHeight);
               hbmMask = CreateBitmap(bm.bmWidth, bm.bmHeight,
                            1, 1, NULL);
               hbmOld = (HBITMAP)SelectObject(hdcBitmap, hbm);
               hbmOldMask = (HBITMAP)SelectObject(hdcMask,
                               hbmMask);

               DrawIconEx(hdcBitmap, 0, 0, hiconOrg,
                  bm.bmWidth, bm.bmHeight, 0, NULL, DI_IMAGE);

               DrawIconEx(hdcMask, 0, 0, hiconOrg, bm.bmWidth,
                  bm.bmHeight, 0, NULL, DI_MASK);

               SelectObject(hdcBitmap, hbmOld);
               SelectObject(hdcMask, hbmOldMask);

               //
               // Create the new mirrored icon and delete
               //    bitmaps
               //
               ii.hbmMask  = hbmMask;
               ii.hbmColor = hbm;
               hicon = CreateIconIndirect(&ii);

               DeleteObject(hbm);
               DeleteObject(hbmMask);
            }

         }
      }

   ReleaseDC(NULL, hdcScreen);
   }
if (hdcBitmap)
      DeleteDC(hdcBitmap);

   if (hdcMask)
      DeleteDC(hdcMask);

   return hicon;
}

A final area to consider when mirroring in Win32 applications pertains to graphics once again. The following will examine what you need to think of when working with image-list controls.

Mirroring and Image-List Controls

A good way of dealing with graphics is to use image-list controls because of the wide flexibility these controls offer. Image-list controls are also used to manage graphics for other controls like tree-view and list-view controls. However, there is one potential issue that can arise. In the case of direction-sensitive bitmaps, if you are using an image list whose contents will be rendered on a mirrored device context, the image list will not have direct access to the image-list device context. On Windows XP, you can use the image-list flags ILC_MIRROR and ILC_PERITEMMIRROR to enable mirroring. If your code needs to run on Windows 2000, Windows 98, or Windows Me with mirroring support, you can use the previous code samples to mirror the direction-sensitive bitmap or icon before adding it to the list. Note, however, that the previous code will allow you to mirror such bitmaps and icons if you are adding these images to the list one by one. If you are adding images to the list more than one at a time, you can tweak the previous code to mirror each element inside the image strip that you are adding to the list. For example, you can modify CreateMirroredBitmap(), so that instead of this line:

BitBlt(hdcMem2, 0, 0, bm.bmWidth, bm.bmHeight, hdcMem1, 0, 0,
   SRCCOPY);

you can have the following code, where Width is the width of each individual image in the image strip that will be added to the image list:

int n = bm.bmWidth / Width, i;
for(i =0; i < n; i++)
{
    BitBlt(hdcMem2, i * Width + 0, 0, Width, bm.bmHeight,
      hdcMem1, (n - i - 1) * Width, 0, SRCCOPY);
}

Specifying the width of each individual image will achieve the same effect on Windows 2000, Windows 98, or Windows Me as if you were to specify ILC_PERITEMMIRROR on Windows XP.

Thus far in this chapter, you have seen ways to leverage APIs, extended styles for windows, and good coding practices for mirroring within a Win32 environment. The following discusses the best ways to go about mirroring Web content. Included are tips regarding directional images, left alignment of text, absolute positioning, and tables.


Mirroring in Web Pages

Applying right-to-left layout to Web content is much easier than it is for Win32 applications. Microsoft Internet Explorer supports RTL rendering of HTML content natively. Toggling between LTR and RTL display in the browser is as simple as adding one attribute-direction (DIR)-to the HTML element. <HTML DIR=RTL> will cause the entire page to display from right to left. This means that the page is rendered as if the upper-right corner were the origin, with the x- axis increasing from right to left, as described earlier. A well constructed page will flow correctly with the DIR attribute set to either RTL or LTR. (If you do not use the DIR attribute, the layout will default to LTR.) There are important differences between placing the DIR attribute in the <HTML> tag and placing it in the <BODY> tag. DIR in the <BODY> tag will affect all visible elements in the document, but will not affect ambient properties (stationary elements). Placing DIR in the HTML tag will set the ambient property of the document to AMBIENT_RIGHTTOLEFT for ActiveX controls. Also, if RTL reading order is set in the HTML tag, the reading order of the document can be changed at run time by setting the DIR property of the document object by scripting.

Note
The RTL layout of Web content is not a Microsoft-specific technology, but rather is part of the HTML and cascading style sheets (CSS) standards. Therefore, it might be considered inappropriate to use the term "mirroring" when talking about enabling HTML pages for RTL layout.

To experiment with RTL rendering, try running the following page in Internet Explorer:

<HTML DIR=RTL>
<HEAD>
<TITLE>Playing with document.dir</TITLE>
<SCRIPT LANGUAGE=javascript>
<!--
function myfunction()
{

   if (document.dir = "rtl")
      document.dir = "ltr";
   else document.dir = "rtl";
}
//-->
</SCRIPT>

</HEAD>
<BODY>
<P>This is some text...</P>
<P>
   <INPUT type="button" value="Switch" onclick=myfunction()>
</P>
<Table border=1>
   <TBODY>
   <TR>
      <TD width=100>Cell 1</TD>
   <TD width=100>Cell 2</TD>
   </TR>
   <TR>
      <TD width=100>Cell 1</TD>
      <TD width=100>Cell 2</TD>
   </TR>
   </TBODY>
</Table>
</BODY>
</HTML>

Clicking the Switch button toggles the display from RTL to LTR and back again. Compare the nonmirrored document in Figure 8-8 with the mirrored version in Figure 8-9. Notice how the text in the mirrored document does not change direction, but does change alignment and flow. For this same document, notice also how the punctuation moves from the right side of the text to the left side, and how the scroll bar on the page does the same. Similarly, the cells in the table become reversed when you toggle directions.

Figure 8.8 A nonmirrored version of the document.
Figure 8.8 - A nonmirrored version of the document.
Figure 8.9 A mirrored version of the document. The text is right-aligned, punctuation and the scroll bar appear to the left of the text, and the previous order of the cells is reversed.
Figure 8.9 - A mirrored version of the document. The text is right-aligned, punctuation and the scroll bar appear to the left of the text, and the previous order of the cells is reversed.

The DIR attribute of an HTML file can also be modified first by right-clicking the content of the page and then choosing the resulting context-sensitive menu's Encoding subitem (such as Right-To-Left Document). (See Figure 8-10.)

Figure 8.10 - The http://www.microsoft.com/globaldev page with the RTL directional attribute set.

The DIR attribute can be used in the following ways:

  • When direction is specified as RTL in the <HTML> tag, the page will be set with the correct extended styles so it displays the page as RTL on a bidirectional-enabled system, and so a vertical scroll bar appears on the left side.
  • When direction is specified as RTL in the <BODY> tag, frames and captions will not inherit the RTL direction.

The following HTML code sets the DIR attribute to RTL:

      <HTML dir="rtl">
      <!-- Or -->
      <body dir="rtl" ...>

Despite the relative simplicity of mirroring Web content (when compared with mirroring Win32 applications), the complex positioning that is applied to HTML elements brings up several issues that you will need to be aware of whenever you are designing Web pages for RTL languages. One of the considerations that you should take into account includes how to properly handle directional images. You should also be aware of the problems associated with forced left alignment of text and with CSS absolute positioning. Finally, you should take advantage of the capabilities that tables offer for rendering RTL text.

Directional Images

Internet Explorer's rendering engine-Mshtml.dll-is enabled to handle RTL layout of Web content. Applying the DIR=RTL attribute does several things. It sets the reading order of the text to RTL, aligns the text to the right, and mirrors Web content such as dynamic elements, tables, and cells. However, applying the DIR=RTL attribute leaves the orientation of stationary elements, such as images, unchanged. This is mainly to prevent flipping of all the images that should not be flipped. Directional images, in contrast, need to be flipped and should follow the same layout as the text and content.

For example, Figure 8-11 shows sample text preceded by an arrow in the LTR document. Once the text is set to RTL, the arrow's original orientation remains the same, whereas it should be reversed.

Figure 8.11 The arrow to the left of the text in the LTR document needs to be reversed in the RTL document.
Figure 8.11 - The arrow to the left of the text in the LTR document needs to be reversed in the RTL document.

In order to supply a single code base that renders correctly in either direction, it is not enough to simply provide alternate images for the bidirectional versions of the site. You must also include script in the Web page that dynamically changes the image depending on the rendering mode being used. To effect this change, you can use a CSS filter for the images on the site. The filter that is installed with Microsoft Internet Explorer 4 and later-known as "flipH"-flips any element horizontally, so that the element displays as a mirrored image. The following is the inline script that determines the direction of the images:

<Script Language=VBScript>
Option Explicit
Dim strFilter
If document.dir = "RTL" Then
   strFilter = "filter:flipH;"
Else
   strFilter = ""
End If

document.write "<IMG SRC='images/arrow.gif' Style='" &
   _strFilter & "'>"

</Script>

Alternatively, in an HTML file, the code would be as follows:

<IMG Style=filter:flipH  SRC=C'images/arrow.gif'>

Forced Left Alignment of the Text

By default all tables, cells, and text in Latin-based script are aligned to the left. Explicitly adding the tag marked ALIGN=LEFT would overwrite the DIR attribute and would prevent the RTL text from being properly set to the right. Thus the ALIGN parameter in this case is superfluous. Because the default alignment for LTR display is left in any case, and because the default alignment for RTL display is right (since the DIR=RTL attribute has been applied), removing the ALIGN attribute completely will cause the page to display as intended. That is, it will be left-aligned for LTR languages, and right-aligned for RTL languages. As a general rule, less is more. If you put into your code only those elements and attributes necessary to produce the effect you need, you will avoid unnecessary headaches later on.

Absolute CSS Positioning

Absolute CSS positioning presents special problems for bidirectional Web sites. Figure 8-12 shows a dialog box after translation to Arabic and application of the RTL attribute. The buttons on the dialog box were placed using absolute CSS positioning. Not only do the buttons overlap the text, but the order of the buttons is fixed; the Yes button is fixed on the left side, and the No button is fixed to its right.

Figure 8.12 A dialog box in which the buttons, which were placed using absolute CSS positioning, both overlap the text and have a fixed order.
Figure 8.12 - A dialog box in which the buttons, which were placed using absolute CSS positioning, both overlap the text and have a fixed order.

The following is the code that used absolute positioning:

<BUTTON ACCESSKEY="Y"
      style="font-family: Arabic Transparent, arial;
         font-size: 10pt; position:absolute;
 width: 58pt;
         height: 19pt; left: 146pt; top: 230pt
"
      onclick="doOK()" >
<u>Y</u>es</BUTTON>

In this case, absolute positioning was superfluous. Had the position attributes been left off the buttons, the buttons would have aligned themselves naturally. Removing the position, left, and top attributes (shown in bold in the previous code sample) from the style on the button causes the button to align more naturally below the text. In addition, allowing Internet Explorer to control the flow allows the button positions to be reversed, which is how they should be in this instance.

Reversibility Offered by Tables

Tables are the best and most reliable way to ensure that your content is correctly reversed when moving from LTR to RTL reading order, provided you do not specify alignment on the table cells. For example, take a look at the following HTML code:

<Table width=100% border=0>
<TR>
   <TD bgcolor="#C0F000"><NOBR>Item 1 </NOBR></TD>
   <TD width=30 bgcolor="#C0F000"></TD>
   <TD bgcolor="#C0F000"><NOBR>Item 2 </NOBR></TD>
   <TD width=30 bgcolor="#C0F000"></TD>
   <TD bgcolor="#C0F000"><NOBR>Item 3 </NOBR></TD>
   <TD width=700 bgcolor="#C0F000"></TD>
   <TD bgcolor="#C0F000"><NOBR>Item 4 </NOBR></TD>
</TR>
</Table>

To summarize, here is a list of general rules for site design abstracted from the information presented in the previous sections that can help ease the process of translation and text reversal:

  • Avoid specifying ALIGN=LEFT unnecessarily.
  • Avoid CSS absolute positioning.
  • Use tables for robust reversibility.

Though not discussed in this chapter, you should also avoid inline styles. This rule is one of the coding standards of the HTML AutoLayout (HAL) documentation, which Microsoft uses internally. (To view the HAL documentation, see the Extras subdirectory on the companion CD, or go to http://www.microsoft.com/technet.) You have seen how to handle mirroring in Win32 applications and in Web pages. The final focus of this chapter is on the best ways to perform mirroring within the .NET Framework.


Mirroring in the .NET Framework

Glossary
  • ADO.NET: Stands for Microsoft ActiveX Data Objects for the .NET Framework. A set of classes that expose data-access services to the .NET programmer. ADO.NET supplies a rich set of components for creating distributed, data-sharing applications. It is an integral part of the .NET Framework, providing access to relational data, Extensible Markup Language (XML) integration, and application data.
  • Web Forms: Generally applies to the design-time technology that enables the authoring of ASP.NET pages in a visual designer.

The remainder of this chapter shows how to mirror Web Forms, Windows Forms, and message boxes. As was true for Web content, the DIR attribute can be used for mirroring text in Web Forms. The RightToLeft property-from the Control class-is valuable for providing RTL support for Windows Forms, whereas the MessageBox class supports RTL and mirroring for message boxes. In the sections that follow, you will learn how to take advantage of this support.

Mirrored Web Forms

In Microsoft Visual Studio .NET, you can create ASP.NET applications by using either Microsoft Visual Basic .NET or Microsoft Visual C# .NET. The guidelines that you should follow are basically the same as those you have just seen for mirroring Web content. When you design Arabic Web Forms pages, the best way to make text flow from right to left is to use the DIR attribute. As with Web content, this attribute is usually placed in the <HTML> tag or the <BODY> tag. Controls and HTML elements on the page then inherit the specified direction.

You can set the DIR attribute at a DOCUMENT object level. All the controls on the form will inherit the same settings. However, the DIR attribute can be used individually with other tags such as <TABLE> and in Web Forms controls, as in the following example, which allows items to be displayed from right to left:

<TABLE dir="rtl" ...>
<asp:TextBox dir="rtl" ...>

Mirrored Windows Forms

Glossary
  • Multiple-document interface (MDI): A UI in an application that allows the user to have more than one document open at the same time.
  • Windows Forms Designer: Provides a rapid development solution for creating Windows applications. It is the locus of visual, client-based forms design. Using the designer, you can add components, data controls, or Windows-based controls to a form.
  • Windows Forms controls: Reusable components that encapsulate UI functionality and are used in client-side Windows applications. Not only do Windows Forms provide many ready-to-use controls, they also provide the infrastructure for developing your own controls.

Windows applications created in the .NET Framework allow you to access operating-system services and to take advantage of other benefits that your user's computing environment provides. Also, you can access data using ADO.NET. Additionally, GDI+ allows you to do advanced drawing and painting within your forms. (For more information on GDI+, see Chapter 13, "Graphics Device Interface Plus.") Finally, your Windows applications can make calls to methods exposed through XML Web services.

Windows Forms can be standard windows, multiple-document interface (MDI) windows, dialog boxes, or display surfaces for graphical routines. The easiest way to define the UI for a form is to place controls on its surface. Forms are objects that have several functions. The objects expose properties that define the forms' appearance, methods which define their behavior, and events which define their interaction with the user. Windows Forms are controls because they inherit from the Control class. This class has a key property for RTL support-the RightToLeft property-which specifies whether the text is displayed from right to left. The form itself supports the RightToLeft property, as do all your controls.

The RightToLeft property takes one of the following values, as shown in Table 8-1.

Table 8-1 Values for the RightToLeft property.

Value | Description
Inherit | The direction in which the text is rendered is inherited from the parent control.
No | The text is rendered from left to right. This is the default value.
Yes | The text is rendered from right to left.

The effect of the RightToLeft property can differ slightly from one control to another, as shown later in Table 8-2 below. In the Form object, when you set the RightToLeft property to Yes, text displayed in the form's title bar is right-aligned and has an RTL reading order.

What about design time? The Windows Forms Designer takes into account the effects of the RightToLeft property on controls. For example, when you set the RightToLeft property of the form to Yes, the property automatically updates the display of the form in the designer, so that the form's caption appears in the title bar right-aligned, as at run time. When the RightToLeft property value is set to RightToLeft.Yes, the horizontal alignment of the control's elements is reversed, yet the elements' alignment values are unchanged. For example, in a TextBox control with the TextAlign property value of HorizontalAlignment.Left, text is displayed right-aligned, but the property value remains HorizontalAlignment.Left. However, if the RightToLeft property value is set to RightToLeft.Yes and if the TextAlign property is set to HorizontalAlignment.Right, the text is displayed left-aligned.

Windows Forms don't support mirroring directly, as they do the RightToLeft property. Instead, you create your own controls to be displayed as you need. You can develop your own RTLTreeview control, for example, which inherits from the Treeview control and changes its style to be mirrored.

On some occasions, you might want to create a basic form with settings and properties such as a watermark or a certain control layout that you will then use again within a project. Each new form can contain modifications to the original form template. The solution is to use Windows Forms inheritance, which is a feature offered with the .NET Framework class library. For example, using inheritance you can build your own mirrored form by inheriting from the System.Windows.Forms.Form class and changing the window style of System.-Windows.Forms.Form.

Table 8-2 on the following page explains how the RightToLeft property affects the Windows Forms controls and which of them need to be mirrored. The common effect of setting the RightToLeft property to Yes on all controls is that the base-text embedding (in other words, the overall direction of text rendering) is assumed to be right to left. Table 8-2 explains how the RightToLeft property affects the Windows Forms controls and indicates which of them need mirroring.

Table 8-2 Explains how the RightToLeft property affects the Windows Forms controls and indicates which of them need mirroring.

Control or Component | Right-to-Left Effect | Require Mirroring
Button | Sets the RTL reading order. | No
CheckBox | The check box is displayed on the right side of the text. | No
CheckedListBox | All the check boxes are displayed on the right side of the text. | No
ColorDialog | Not affected; depends on the language of the operating system. | No
ComboBox | Items in combo-box control are right-aligned. | No
ContextMenu | Appears right-aligned with RTL reading order. | No
DataGrid | Appears right-aligned with RTL reading order. | No
DateTimePicker | Not affected; depends on the language of the operating system. | No
DomainUpDown | Left-aligns the up and down buttons. | No
ErrorProvider | Not supported. | No
FontDialog | Depends on the language of the operating system. | No
GroupBox | The caption of the GroupBox is displayed right-aligned and allows all the contained controls to inherit this property. | No
VScrollBar | Displayed on the left side rather than right side of scrollable controls. | No
HScrollBar | Start with the scroll box right-aligned. | No
ImageList | Not required. | No
Label | Displayed right-aligned. | No
LinkLabel | Displayed right-aligned. | No
ListBox | Items in list box control are right-aligned. | No
ListView | Sets the reading order to RTL, while the elements stay left-aligned. | Yes
MainMenu | Displayed right-aligned with RTL reading order, during run time only and not in design time. | No
MonthCalendar | Not affected; depends on the language of the operating system. | No
NotifyIcon | Not supported. | No
NumericUpDown | Up and down buttons are left-aligned. | No
OpenFileDialog | Not affected; depends on the language of the operating system. | No
PageSetupDialog | Not affected; depends on the language of the operating system. | No
Panel | Allows all the contained controls to inherit this property. | Yes
PictureBox | Not supported. | No
PrintDialog | Not affected; depends on the language of the operating system. | No
PrintDocument | The vertical scroll bar becomes left-aligned and the horizontal scroll bar starts from the left. | No
PrintPreview | Not affected by this property. | Yes
PrintPreviewDialog | Text displayed in the title bar of a form is right-aligned. | No
ProgressBar | Not affected by this property. | Yes
RadioButton | The radio button is displayed on the right side of the text. | No
RichTextBox | Control elements that include text are displayed from right to left with RTL reading order. | No
SaveFileDialog | Not affected; depends on the language of the operating system. | No
Splitter | Not supported. | No
StatusBar | The text in the panels aligned to right with RTL reading order. | Yes
TabControl | Not affected by this property. | Yes
TextBox | Displays text from right to left with RTL reading order. | No
Timer | Not required. | No
ToolBar | Not affected by this property. | Yes
ToolTip | Sets the RTL reading order. | No
TrackBar | The scroll bar or track bar starts from the right side of the screen. | No
TreeView | Sets the RTL reading order only. | Yes

Message Boxes

As its name suggests, the MessageBox class represents the message box; the Show method of this class displays a message box that can contain text, buttons, and symbols that inform and instruct the user. The MessageBox class fully supports RTL reading order and mirroring. The Show method of the MessageBox class takes constants defined in the MessageBoxOptions enumeration as parameters. These constants include RtlReading and RightAlign, both of which enable the message box to display bidirectional text. The following code illustrates how to display an Arabic message box:

[Visual Basic ]
MessageBox.Show("  ...",
"  ",
   MessageBoxButtons.OK,
   MessageBoxIcon.Question,
   MessageBoxDefaultButton.Button1,
   MessageBoxOptions.RtlReading Or MessageBoxOptions.RightAlign)

RtlReading is responsible for ensuring that the text in the message box is displayed with RTL reading order, with mirroring applied. RightAlign allows the text to be right-aligned only, but does not enforce RTL reading order or mirroring in the message box.


Summary

If you are planning to localize your application into a language that has an RTL reading order such as Arabic or Hebrew, both the text and the UI elements will need to be laid out from right to left. The mirroring technology gives a perfect RTL look and feel to an application's UI, whether you are dealing with a Microsoft Win32 application, a Web-based application, or an application within the .NET Framework.

For Win32 applications, you can either enable mirroring in resources or in code. The first approach is a good one when you want to mirror old components for which the source is not available or is no longer used; it is also beneficial when you want to conduct primary testing of your applications to see if they support mirroring. However, this method does not allow you to apply mirroring to a specific window or control, since all windows and their controls will be mirrored automatically. In addition, mirroring in resources is merely a workaround used mainly for primary testing; it is not a Microsoft-supported development technique.

A better method is to enable mirroring in code. By doing so, you can identify exactly what you want to be mirrored. You also have the liberty of determining whether you want to mirror per process, per window, for dialog resources, or per device context. If you activate mirroring per process, all newly created windows are automatically mirrored, though existing windows are unaffected. Activating mirroring per window allows you to deactivate mirroring for particular windows. By activating mirroring per device context and by calling various GDI functions, you can control whether a bitmap or icon should be mirrored, since the device context of a mirrored window is normally mirrored by default.

In terms of mirroring Web content, Internet Explorer's rendering engine is mirroring-aware. By applying the DIR=RTL attribute in your HTML code, you can set the reading order of the text to RTL, align the text to the right, and mirror dynamic elements, tables, and cells. When working with Web content involving RTL languages, avoid specifying ALIGN=LEFT unnecessarily, avoid CSS absolute positioning, and use tables for robust reversibility.

Within the .NET Framework, use the DIR attribute in the <HTML> tag or the <BODY> tag for Web Forms. For Windows Forms, provide RTL support with the Control class and the RightToLeft property. Finally, for message boxes use the MessageBox class, which fully supports RTL reading order and mirroring. These guidelines will help you create a fully localizable UI for RTL languages.


Chapter 9: Content Localizability Guidelines
Glossary
  • User assistance (UA): Refers to any form of documentation (including printed documents, online documents, and Help files) that corresponds to a particular product.
  • Content recycling: Reusing content in localization. Content recycling saves time and money because the content is researched, presented, edited, reviewed, and translated only once.

When building a product, you have to consider the software and its corresponding documentation as one entity. A world-ready product does not just mean world-ready software; all the supporting documentation must also be made world-ready.

For documentation to be world ready (that is, globalized and localizable), it must meet the following requirements:

  • The content must be kept as simple as possible and should follow basic writing style principles such as those found in The Chicago Manual of Style.
  • The content (whether text, art, or multimedia) must be culturally acceptable, politically correct, and inoffensive to an international audience, regardless of whether the material is to be localized. There are several reasons for this-even for products that are not localized, Canadian, British, and Australian readers (at a very minimum) read the same U.S. English version of the documentation. Moreover, readers often look at the English version of the Help files.
  • The content must follow consistent patterns. Content recycling tools rely on documentation that adheres to strict style guidelines and consistency rules across successive releases of a product. This consistency speeds up content translation and helps control localization costs.
  • No matter what locale is selected, the Help system must have the same functionality.

Of course, not all content will be localized. For instance, no product is localized into all the languages of the world. Nor are all platforms of a product (such as Microsoft Windows XP) localized into the same set of languages. In addition, many companies throughout the world use software that has not been localized, and many books written to assist in using the software are available only in English. Nevertheless, even if your content remains in English, it still must meet the requirements just listed.

UA content that is globalized properly has two advantages. First, it is easier to be understood in its intended local marketplace as well as in markets that are culturally different but that read the same language. Second, it is ready to be localized with a minimum of difficulty-in other words, the content becomes localizable. Whether your content involves printed or online text, art and multi-media, or a Help system, you should adhere to the best practices presented in this chapter to speed up the localization process and to minimize the potential for inadvertently confusing (or offending) the reader.


Keep Content Simple

Localization relies heavily on the original content being world-ready and culturally appropriate for translation. However, whether content is localized or not, it must be kept simple. Here are some guidelines to help you do so.

  • Be as concise as possible. Choose words that are direct and to the point. This editorial approach benefits native speakers of your language and international readers.
  • Use the active voice. This verb form shortens your sentences and is a more familiar construction for most international readers.
  • Avoid using convoluted and long-winded adverbs and expressions. For example, do not use "in order to" when you can use "to," and do not use "entirely sold out" when you can use "sold out."
  • Use simple verb forms. For example, use "click..." instead of "you should be clicking..."
  • Choose single-word verbs over multiword verbs. For example, use "Lock the system" instead of "Lock down the system." Use "select" instead of "make a selection."
  • Finally, make sure that you use correct grammar. For example, the spelling of the irregular possessive form is "its," not "it's" ("its features," not "it's features").


Follow Basic Writing Style Principles

The following guidelines will help ease localization and increase comprehensibility.

  • If possible, use words that have a precise and confined meaning. For example, use "Install the application" instead of "Set up the application." Simpler, more unambiguous terminology makes your content easier to understand. Using short, concise paragraphs also helps reduce the risk of misinterpreting what you are trying to say.
  • Use the same term to express a specific meaning or concept. For example, do not use "remove" and "delete" interchangeably.
  • Include "optional" words (words that are often omitted) in sentences. Omitting such words might increase the chance of misunderstanding your content. For example, "You can change the Admin.tpl and the Admin.inf files using the Template utility." can have two different interpretations: "You can change the Admin.tpl and the Admin.inf files that use the Template utility." or "You can change the Admin.tpl and the Admin.inf files by using the Template utility." Either sentence could be correct depending on the context.
  • Use consistent capitalization for proper nouns and for user interface (UI) terminology. Doing so will help standardize these items for the localizer.
  • Use "close" or "tight" punctuation (despite the current tendency to use less punctuation). This helps reduce confusion for the translator and the reader and breaks complex sentences into easily digestible parts.
  • Do not omit articles ("a," "an," and "the"), and use correct capitalization and punctuation. Following these rules will help remove ambigu ity from-and add clarity to-your content, which reduces the burden on the reader and also reduces the likelihood of mistranslations.
  • Avoid using alphabetical lists (with tables, glossary entries, Microsoft MS-DOS commands, keyboard shortcuts, or any other type of reference) that cannot be regenerated automatically. Alphabetical order breaks during localization of the text because entries have to be reordered, a very time-consuming task. Lists can often be extremely long, as in the case of reference glossaries. Therefore, limit their use, unless you can provide localizers with some type of sorting utility.
  • Do not use a noun as a verb. Use "Establish contact with the company." instead of "Contact the company." Use "This feature makes your operating system current." instead of "This feature currents your operating system." The following sentence is an even better choice, since "up to date" is more precise than "current": "This feature makes your operating system up to date."
  • Avoid words that have opposite meanings. For example, the term "sanction" can mean "to approve" or "to punish."


Respect Cultural and Local Sensitivity

The following guidelines will help ensure that your content takes into account the wide spectrum of cultural and local differences that exist among international readers.

  • Remove any text that might raise cultural-sensitivity issues in order to create neutral, culture-independent content. Do not use slang and colloquial expressions; your international readers usually cannot distinguish the idiom from the literal meaning, which will make it harder for them to understand your content. Moreover, software localizers and translators can make similar mistakes as they localize the country-centric content.
  • Avoid cultural-specific metaphors. A metaphor often carries with it some cultural connotations that might not have any relevance outside your country. A good example is the word "touchdown." Using this word to convey the meaning of an achieved goal won't be understood in many countries because American football, from which that term is borrowed, is not necessarily a familiar sport.
  • Avoid humor. It adds neither value nor clarity to international content.
  • Stay away from ambiguous words and jargon. Jargon is technical terminology-such as "dogfood" (sometimes used for software that is not fit for public consumption yet is utilized for internal purposes)-tailored for a par ticular business or profession. Jargon is almost never appropriate in globalized content.
  • Spell out a term's acronym or abbreviation the first time the term appears, followed by the acronym or abbreviation in parentheses. Then, in the same topic or unit, you can just use the acronym. You should spell out the acronym at least once in every Help topic because readers randomly browse Help topics. Some abbreviations are accepted and should not be spelled out, even at first mention. (For example, you can use "St. Petersburg," as opposed to "Saint Petersburg.") Also, some technical abbreviations do not need to be spelled out at first mention because they are so widely used and understood (for example, TCP/IP, DHCP, or TAPI). Localization specialists can tell you whether acronyms are familiar to foreign users.


Write for Easy Recycling and Reduced Localization Costs

You should author your content as though it is going to be localized. Using this rule in the initial content creation stages allows you to design proper content and graphics with local markets in mind, which will ease localization should the plan change. Most content localization is performed using content recycling tools that are based on existing translated content; costs for localization projects that use text recycling tools are much lower. There are several guidelines you should follow to enable easy recycling and to reduce costs.

Only change the content in a product upgrade if the justification is more than cosmetic. Changing the content from a previous version results in eliminating the exact matches in the already translated content, which means that the content has to be manually translated. Do not make unnecessary changes. For example, do not replace "The following tools are available with Microsoft Windows 2000." with "The tools listed below are available with Microsoft Windows 2000." If you do, loss of recycling will result, without any value added to the content.

Also, be as consistent as possible. For example, either use "Click the icon." or "Single-click the icon." rather than using both phrases interchangeably. Consistency increases the number of repeated sentences and segments in the content. These sentences and segments can be batch translated all at once by most content localization tools, and the manual translation of one occurrence takes care of all other occurrences automatically. The results are a faster localization cycle, reduced localization costs, and a higher quality in the U.S. content and in the localized content. Finally, remember these basic rules.If you can remove a sentence because it does not add clarity to the content, do it. If you can cut words from a recycled sentence because they are superfluous, do it. If you can preserve recycling because the legacy content is still accurate and conforms to standard style guidelines, do it.


Respect Cultural Sensitivity in Art and Multimedia

For art, as well as for text, you need to "think globally and act locally." Before using any image, consider whether that image might be offensive or confusing to another culture. Not all images are universally meaningful.

Clip art is often inappropriate for international customers. Even if the graphic has been reviewed in all the local markets before use, localization teams should always be allowed to either delete the picture or replace it with clip art that is appropriate for localized versions. Hence, either use clip art sparingly, or do your homework on locally-correct clip art. For example, do not use clip art to represent the concept of "currency," since currency has different graphical representations from country to country. Here are some examples of questionable clip art. (See Figure 9-1.)

Avoid using country-centric graphics and icons, such as mailboxes, road signs, or even forks and knives, in your content. These images represent concepts that are obvious signs of nationality; globalization entails making your content as culturally and politically neutral as possible, so it does not confuse or offend readers who are not familiar with the concepts of your culture or language. Here are some examples of questionable icons. (See Figure 9-2.)

Figure 9.1 Questionable clip art for representing a generic concept.
Figure 9.1 - Questionable clip art for representing a generic concept.
Figure 9.2 Questionable icons that use an American football icon to represent sports, and a fork and knife icon to represent a restaurant.
Figure 9.2 - Questionable icons that use an American football icon to represent sports, and a fork and knife icon to represent a restaurant.

Understand the consequences of using multimedia. For example, for UA content that is not localized, U.S. English accents might not be comprehensible to a foreign audience. In UA content that does get localized, translators might not understand what is being said clearly enough to translate successfully.


Design the Help System with Global Functionality

If your UA content ships as a Help system, remember that the Help system is software. Help systems often use scripting to add features to the Help topic, such as demoted and expanded lists, shortcut links that start applications, and so on. The scripting has to function independently of the locale that is selected. Make all of the templates used to create your documentation "core" and global from the beginning. For example, make sure that the following elements are in the template that you use as the basis for authoring your content:

  • The directionality attribute, which can be changed to create a right-to-left document; (<HTML DIR=RTL>)
  • Explicit definition of the character set-for example, charset=attribute, which is used in the HEAD portion of the HTML document and tells the browser how the text in the document has been encoded

If you are authoring Extensible Markup Language (XML) content, follow the World Wide Web Consortium's (W3C) guidelines on XML at http://www.w3.org.


Summary

When creating a product that is world-ready, you must make sure its documentation is globalized and localizable. Keep the content of your documentation simple, and follow basic writing style principles. Also, make sure your content (whether text, art, or multimedia) is culturally acceptable, politically correct, and inoffensive to an international audience, regardless of whether the material is to be localized. Write content that can be easily recyclable to save localization costs and to make localization easier. Avoid using country-centric graphics and icons so that you do not offend or confuse the reader. Finally, design your Help system so that it has global functionality. For instance, make all of the templates used to create your documentation "core" and global from the beginning. The scripting used within Help topics needs to function regardless of the locale that is selected.

Adhering to the guidelines in this chapter will have numerous rewards. It will improve the understanding of your content, not only by native speakers of your language, but also by international readers and localizers-which, in turn, helps to make it local izable. It will cut down on localization costs and, finally, it will contribute to having a product that is world-ready.


Part IV: Localization

So far this book has concentrated on how to design and create a world-ready application—one that is globalized and localizable—which is the first key process of creating an internationalized product. This part of the book focuses on the second key process of internationalization—localization. The concepts and practices discussed in Part III,"Localizability," and Part IV, "Localization," work in tandem to ensure that your software product's code, resources, and content are fully adapted for a particular market. While the chapters within Part III helped you prepare your application for both software and content localization, Part IV takes this effort one step further.

Localization is a complex process of translation, adaptation of cultural, political and graphic elements, troubleshooting, and if localizability is not done correctly, re-engineering of the original product. Some of the many elements that are modified in software and content localization include text, layout, graphics, keyboard shortcuts, fonts, character sets, and locale data, in addition to the product's build process and packaging. Chapter 10, "Localization," identifies areas to be aware of in order to expedite localization of these elements. It also pinpoints the criteria that good localization tools should meet, shows the people, tasks, and skills that comprise a localization team and process, and gives recommendations regarding scheduling and localization outsourcing.

The material within Part IV is intended as a way to acquaint you with the localization process, so you can tailor your design and development decisions accordingly. It is not meant to teach localizers how to do their job, which could comprise an entire book in itself. For this reason—and also because globalization, localizability and localization are separate but connected functions in creating international software—Part IV consists of one chapter.


Chapter 10: Localization

If you buy an English translation of Cien años de soledad, the novel by Gabriel García Màrquez, you do not expect the story to be transferred to the Midwest of the United States. Instead, you know that this is a translation of a book that was originally written in Spanish and that the story takes place somewhere in Colombia. Other than the fact that the language is different from that of the original version, you want the translated story to be identical to the original-taking place in the same location, with the same cultural background. Even when translated, the novel's linguistic nuances should be as close as possible to those in the original text that Garcia Màrquez wrote.

The opposite is true when it comes to software. Most people would not necessarily care that the software they are using might have been developed in a small town in the Pacific Northwest of the United States, for instance. Instead, people expect software to give them the impression as if it was developed in their native tongue and culture-Japanese, French, Arabic, or German. Thus software localization is geared toward adapting software not only to the language but also to the culture of the target market. Translation is only part of the job.

Imagine you buy some educational software for your eight-year-old, and you notice that the children in the software program sip a lot of coffee. While many parents in Brazil might find that behavior absolutely natural, most parents in the United States probably will not. Thus if you want to sell your localized product in the U.S. market, you might want to replace the coffee with some kind of soft drink. However, in a localized version for the European market, soft drinks might not be the best choice either, since many European parents consider coffee and soft drinks to be equally unhealthy for their children.

Those who are involved with software localization need to consider these sorts of cultural differences. In contrast, when translating a book or a movie, there is no need to account for cultural variance. If parents don't like the fact that the children sip coffee in a product, they simply need to find another product. The translator won't change the cultural context for them.

Yet another consideration is the geopolitical requirements that need to be addressed for the successful localization of a software application. For instance, in Germany too much bloodshed in a children's game could easily lead to the product being banned altogether. Some terms might be politically sensitive in the target language, even though they are appropriate in the source language.

Indeed, localization does entail a vast amount of translation-including all of a product's text, menus, dialog boxes, buttons, wizards, online Help, printed documentation, packaging, and CD labels. Multimedia files need to be translated, too, and if the product contains videos, the spoken text needs to be carefully re-synchronized as well. However, again, beyond merely translating the language, localization also needs to adapt the product to the particular locale in which it will be used. Currency, address, number, and date formats need to be changed; the sort order possibly needs to be adjusted; and for many Asian languages the fonts and the font size have to be changed, too. Arabic and Hebrew require right-to-left (RTL) layout of not only the text, but of the whole user interface (UI), including buttons, menus, and dialog boxes. (For more information on RTL lay-out, see Chapter 8, "Mirroring." ) As a result, if you are going to localize into certain Asian languages, or into Arabic or Hebrew, you must design your software to support culture-aware features right from the start. The final localized version of the original software product should look and feel as if it had been designed in the user's home country.

The locale for which you adapt your product can be a country or a region where numerous languages are in use, but where there is still a common language and culture. For instance, Mexico has many indigenous languages such as Náhuatl, Huave, Totonaco, and many others, even though the official language of Mexico is Spanish. The localized product can also be marketed in a country where the language of the localized product is not the primary language used in that country. In addition, one version of the localized software can be used in more than one country, provided these countries share the same language. For instance, the French language is used in more than 53 countries, and the Arabic language is used in more than 23 countries.

Now that you have seen an overview of issues that are pertinent to local-ization, this chapter will examine certain specifics such as what elements are typically localized. It will also discuss what capabilities localization tools should have, who comprises a typical localization team, and what constitutes effective localization management. Part III,"Localizability," divided localizability into two categories: software localizability and content localizability. Similarly, this chapter views localization in terms of both software localization and content localization.


Localization Elements

The development team should supply the localization team with a single binary that works well for all different locales. The localization team will expect that the software has been designed and created to meet locale-aware and localizability standards. (For more information, see Part II, "Globalization," and Part III,respectively.) The UI elements should be stored in the resource files of a program or in some other resource repository. If the development team has not done this, localizers should ask the development team to put the resources in a standard format, so that localization tools can handle them. (For more information on storing UI elements, see Chapter 6, "Multilingual User Interface (MUI)," and Chapter 7, "Software Localizability Guidelines.")

There are many elements that are modified in both software and content localization. These elements include text, layout, graphics and multimedia, keyboard shortcuts, fonts, character sets and locale data, as well as the product's build process and packaging.

Text

Text translation is the most important aspect of localizing the software UI. The translation should communicate in language, phrasing, and vocabulary that is natural as well as accurate. Moreover, the language should be clear, familiar to the user, and appropriate for the culture of the target country. Additionally, it should follow conventions used in the target country such as customary symbols, punctuation, formatting, and typography. Title bars, menus, dialog boxes, buttons, messages, and tool tips are some common UI controls that have text. Help files and user manuals typically require heavy translation efforts.

Translation does not always result in a one-to-one correspondence between the source and target language. A single word in English can have multiple translations in another language. For instance, the word "supply" in English can have varying translations in Spanish, depending on whether the word is used as a verb or as a noun. Even within the verb and noun categories themselves, the Spanish translation can differ, depending on the particular context. On the other hand, certain English words might have only a single meaning in another language. In addition to semantic variations among words from one language to another, adjectives and articles sometimes change spelling according to the gender of the nouns. Therefore, when re-using a text string in multiple places, localizers need to make sure that the string is translated in such a way that the text is correct for a given context. (For more information, see Chapter 7.)

Consistent and precise terminology throughout the software application helps the user to learn the application faster and makes the translation time shorter. For example, "traveling" and "travelling" are both correct in English, but it is better to use one variation of the word in the product. Localizers should do the same for the localized text. Having consistent terminology in the original-language product is particularly important if the localization team uses automatic translation tools because the utility might not translate the variation in the same way. (For more information, see Chapter 9, "Content Localizability Guidelines.") Consistent and clear phrasing of the original text makes localization easier and prevents confusion for localizers, who might interpret the text differently from what was meant. Localizers should avoid "sublocale" idioms, colloquialisms, or metaphors, since these might be meaningless-or even offensive-among cultures that use the same language.

Some locales share the same language yet use different words for the same object. For instance, the word "computer" is "ordenador" in Spain and "computador" in Mexico and Puerto Rico (both of which are masculine in gender). The word "computadora" is used in the rest of Latin America (and the word's gender changes from masculine to feminine). To resolve such issues, if there is one translation that is considered acceptable for all locales, localizers should use this particular wording.

In general, the text (as well as the graphics and other elements) of the localized software product should not be altered to reflect locale-specific geopolitical views because this could cause inconsistency in the localized product. Nonetheless, sometimes there could be undesirable consequences resulting from a localized product that is not adapted to meet a locale's geopolitical views. If the views conveyed in the unaltered product are going to offend the target market, this will affect sales in that market. Such a situation would warrant alteration of the localized product. For disputes over national borders, maps, country names, international or political organizations and leaders, or any other politically sensitive subjects, consult an expert on geopolitical issues for advice. Then make any necessary adjustments in the original product. (For more information on political content, see Chapter 7.)

Layout

Another element that requires localization is the UI layout. The localization process changes the UI layout because the length of the translated text usually increases from its original length. This text-stretching affects the size of the UI controls that display the text and potentially necessitates relocating and resizing the UI controls. The best way to ensure that the UI has a consistent look and feel throughout different localized versions is for the development team to design the UI with resizing-and other required locale variances-in mind from the beginning. In turn, the localization team should try to minimize any changes to the UI, unless these changes are absolutely necessary.

In addition to text that increases in length when it is translated, font size can differ among language groups. For instance, East Asian languages usually display text in a larger font size than many other language groups. As a result of the change in font size, the system expands the UI vertically. Edit boxes, static text, and button controls within the East Asian localized version of the product usually result in larger vertical spacing than those within products from other language groups. Bidirectional languages like Arabic and Hebrew require mirroring of the screen layout to accommodate their RTL reading order. This requires mirroring the layout of menus, text, dialog boxes, and edit boxes.

Graphics and Multimedia

One of the advantages of graphics is that they can communicate more universally than text, provided they are not culture-specific. Graphical UI components of the software product might need to be revised for the international audience. Use images that are not geared toward a particular culture or locale, and avoid including text within graphics. These practices will help minimize the localization of graphics, which is very expensive and often causes the shipping of the product to be delayed.

You will also need to retake UI screen shots for the localized versions of the product. Icons that have a particular directional orientation-such as Undo and Repeat-will need to be localized for bidirectional languages in order to be locale-aware. (For more information, see Chapter 8.) The same concept applies to music and video. For instance, suppose you want to use music to indicate that the computer is calculating an equation. Because the famous U.S. game show "Jeopardy" uses a particular musical theme to indicate that a contestant is thinking of an answer to the game-show host's question, you decide to include the music in your application. Unfortunately, this effort will not necessarily be understood in other countries.

The localization team can give feedback about the graphics and multimedia components used in the product to ensure that they are suitable for all locales. Even if it were not expensive to design custom graphics for each language, having different images from one language to another can confuse the user who works with more than one language version.

Keyboard Shortcuts

Keyboard shortcuts might need to be changed according to the translated text that contains these shortcuts. For example, the F key is used as the shortcut for "File" in English. This shortcut needs to be modified during German localization to the D key because the same word in German-"Datei"-does not have the "F" character in its name. Chinese, Japanese, and Korean do not have a Latin-based script that allows for single-key shortcuts. For this group of languages, a common practice is to keep the source language's original Latin-based shortcuts such as " [image] (F)". When there are standard shortcuts for common functions in localized products, localizers should use these shortcuts and not create new ones.

Fonts

Localized products might have different UI fonts than the original-language products. The font name, size, and style in dialog boxes and documents can be changed to customize the final UI based on user preference. Localizers will use the best font settings according to the font capabilities of the particular target language.

Locale Data and Character Sets

Formats of dates, time, currency, measurements, and telephone numbers vary from one locale to another. Again, localizing the software to suit an international audience requires using the appropriate format for the particular locale. You can change a format programmatically by using the globalization infrastructure provided in Windows operating systems and the .NET Framework. (For more information, see Chapter 4, "Locale and Cultural Awareness." ) Some instances of locale-data values can be stored as part of the resources to initialize variables like the default character set or the default date format. If the development team does indeed want to store these values, localizers should change the locale-value strings according to the format of the target locale.

Build Process and Packaging

It is a good idea to use the same build process for the localized product as for the original product. Still, there are some binaries that need to be included in or excluded from the localized product for specific markets, which can alter the build process for the localized product. The localization team should work with the development team to set up an automatic procedure for handling changes, and should make this procedure part of the general production process. Localizers need to make sure all text, controls, and menu items that refer to locale-specific features are changed accordingly.

The basic product design for the localized product should be the same as for the original product. However, product packaging might require a certain amount of local ization so that it appeals to international customers. For instance, the localization team should change any reference that is made to locale-specific features, such as contact phone numbers and addresses. Another important aspect of product packaging is the End User Legal Agreement (EULA), which must be modified according to the laws of the country in order to sufficiently protect the manufacturer.

An important component of localization involves tools that will simplify the localization process. While some people might be able to add up ten numbers in their heads, using a calculator is usually a quicker and easier method of achieving-what should be-the same result. Similarly, the proper localization tools will help expedite the localization process and will handle many of the common tasks that localization entails.


Localization Tools

There are several ways to localize software product files. One method is to edit the original resource files before the compilation process takes place. A second method is to localize the final compiled binaries of the product, and then to either save the localized resources into the same binaries or into a separate localization database. Most localization tools are capable of handling either or both of these methods. Regardless of the localization method you choose, using the proper resource editor tools can make the localization of product resources quick and easy. Effective localization tools should also be able to create the localized version of the corresponding original file. This original file can be a resource file or a compiled file, depending on the localization team's choice of tools and localization process.

There are many good localization tools that are commercially available. Before choosing one, be aware of the various criteria that a localization tool should meet. Besides the ability to translate the string resources within the localization tool, a few other tasks the tool should perform are to:

  • Change the names, sizes, and styles of the UI fonts.
  • Resize, move, and hide controls as necessary.
  • Change the encoding of the text.
  • Replace graphics and icons with localized ones.
  • Modify keyboard shortcuts.

A tool's ability to track changes that are made to the original resource from build to build is another very useful feature for localizers, since they can see what is updated between builds. This change can involve a string resource, a font size, or a new icon that has been added to the original product files. On the other hand, it is also useful for a tool to track resources that have been localized, as well as who did the localization, and other pertinent information. These types of statistics and the percentage of localization work completed help the localization manager track and plan the activities of the localization team.

The localization tool should be able to extract and edit a software project's different resource-file formats, which can include INF, Microsoft Win32 resource files, Extensible Markup Language (XML), and many other file formats. The localization team needs to choose the appropriate localization tool in accordance with the resource-file format used in the project.

Another requirement of a good localization tool involves its handling of localization glossaries. (For more information on localization glossaries, see "Localization Outsourcing" later in this chapter.) The tool that creates these useful glossaries helps the localizers look up the translation of English strings within the glossaries, thus providing a consistent translation. Recycling the localization glossaries allows automated translation of any new string resources in the product that resemble previously translated strings, a practice which will significantly reduce localization costs. Finally, the tool should provide a way to add instructions on how to localize the resources; this will make localization easier and more efficient.

Of course, effective localization results not only from having the proper tools but from having the right people to perform all necessary tasks in the localization process. The following section identifies the team members on a typical localization project and examines each member's roles and responsibilities.


Localization Team

A large localization project requires a team that consists of a program manager, one or more localization engineers, and many localizers. The program manager should coordinate between the development team and the localization team, making sure that developers use code practices that are conducive to localization. (For more information on these practices, see Chapter 7.) The program manager should also deal with the external teams that do localization and, of course, manage the internal localization process.

Localization engineers are responsible for handling the localization database and maintaining the proper localization instructions in the commenting model, if this model exists. For standardization purposes, localization engineers should prepare localizability and localization guidelines for the different product teams to educate them about the localization process. In addition, they need to make sure that localizers have enough information to translate the string resources correctly. Once localization is complete, localization engineers also need to verify that the localization instructions were implemented. Localizers translate strings and make other localization changes such as changing the layout of the UI, localizing graphics and multimedia, adapting the build process, and redesigning packaging. Since technical experience is such an integral part of localization, localizers should possess technical resources in addition to language skills.

The Localizer's Technical Knowledge

Glossary
  • Commenting model: A way for the development team to pass along localization instructions to the localization team. This information might include whether strings can be localized or whether limitations on the string length exist, for instance, or can just consist of general comments added to the resource files.

Localizing an operating system is somewhat different from translating Shakespeare. While the latter requires more than excellent language skills, the localization of an operating system definitely requires extensive knowledge about the product and related fields-besides some solid language skills in both the source and target language. It's hard to translate a sentence like "Precreate CR for domain %s1 (a DN) allowing server %s2 (a full DNS name) to dcpromo." when you have no idea at all what a CR or DN could possibly be. Thus, a Ph.D. in linguistics is not necessarily the right qualification for this kind of task, though it probably helps to produce grammatically correct sentences in the target language. Without some in-depth knowledge of the product, a localizer won't be able to make sense of the source text, and thus won't be able to translate the text adequately in the target language. The technical level, however, can vary widely. For the localization of a database application or an operating system, technical skills and knowledge of the product are essential. On the other hand, for the localization of a children's game that contains a lot of spoken text, language skills are much more important, and there's still enough time to get familiar with the product while working on it.

Comprehensive understanding of the source text, however, is only one reason why technical skills are as important as language skills for localization. The localizable resources of a product are extracted from the resource files and are imported into a localization database. At the very least, the resources are placed into some kind of table format to make them accessible to localization. As a result, the individual resources are usually localized without any indication of the context or how they are used in the actual product at run time. As mentioned previously, since the translation in many cases and in many languages can vary widely depending on the context, it is important for the localizer to know how a string is used in the product and what the context is. If the surrounding strings don't provide any clues, the only way to find out how the text is actually used at run time is to install the original-language product and click through the application until the string shows up. Particularly in the case of error messages, that might be not an easy task, since a certain scenario has to be reproduced in order to bring up the error message.

A localizer with the appropriate technical skills will be able to correctly perform the fundamental task of translating functionality-sensitive content; this accuracy is crucial, since a mistranslation could break the product. The commenting model described in Chapter 12, "Testing Localizability with Pseudo-Localization," supplies localizers with valuable information to prevent functionality from breaking. For example, as stated, this model can provide information about the maximum string length. The localizer should heed these instructions and should avoid exceeding this limit to prevent buffer overrun. Strings that are blocked from localization must remain untranslated. There might be special localization instructions to change certain functionality-sensitive strings in accordance with the target language. For example, a font name that exists in the translation table and that is used to display text on application dialog boxes needs to be translated according to the conventions used in the target language. The technical localizer will know which font name to use for a specific language. The same will apply to other string types like font sizes, locale values, and locale-specific file names.

Likewise, the localizer needs to be able to test the localized product in order to check if the translations are appropriate. Consistent translation lookup tables are useful to ensure correct translation of the text that has content dependency. These lookup tables can be created as part of the localization process and can be reviewed after translation is finished.

In addition to using good tools and having localization team members who perform defined functions, you should also have some basic process guidelines in place that will enable localization to run smoothly. The following section discusses some basic areas that you should consider.


Establishing Localization Guidelines

High localization costs are incurred if the localization process is not run effectively. Establishing localization guidelines can increase the quality, accuracy, and user-friendliness of the international product version. Moreover, it can significantly reduce the cost of localizing your application into different languages. Establish clear guidelines for ensuring effective localization, for scheduling international releases, and for localization outsourcing.

Ensuring Effective Localization

Glossary
  • Pilot language: A language that is localized before other languages; this language is used to define the localization process and to test for localizability issues.

The initial product specifications should consider localization as part of the product cycle. Delaying localization to the end of the cycle can delay the shipping date because of any unexpected localization problems that require code changes. Localizability testing should start once the product functionality is stable. Localization for pilot languages or pseudo-localization should start once the software functionality is stable. Pilot-language localization is a good approach for large products that are localized in many target languages to speed up localizability code fixes. Localization for other languages should start once the UI is stable. A UI freeze is recommended before shipping, with enough time allotted so that the localization and user assistance (UA) teams can finish translating the software and Help files on time.

From a language standpoint, avoiding grammar and spelling mistakes in the original software text reduces the localization team's workload when those mistakes are corrected. Corrections will show up as a change to the localization team and will therefore need to be fixed again in the localized version. For the same reason, avoiding cosmetic changes in the content that do not add value to the meaning will speed up localization.

Scheduling International Releases

The product-release schedule can differ from one market to another according to market needs. Ideally the localized product should be shipped on the same date as the original-language product, or with a delta less than 30 days. This concept is known as "simultaneous ship," or "sim ship." (For more information on sim shipping, see Chapter 1, "Understanding Internationalization," and Chapter 2, "Designing a World-Ready Program." ) However, shipping on the same day usually does not happen because localization, documentation printing, and package design continue after the original product has shipped. Figure 10-1 shows a typical time line for the development and shipping of a product. The product team and the international marketing team should decide what the delta between the shipping dates for the original product and the localized product should be. In Figure 10-1, Tier-1 languages refer to localized language versions that are shipped to the most important target markets; Tier-2 languages refer to language versions that ship to secondary markets, or are versions for which only a partial localization is planned. Localization for Tier-2 languages usually starts somewhere around the UI freeze of the original product.

Figure 10.1 Localization time line for international production.
Figure 10.1 - Localization time line for international production.

Localization Outsourcing

Most of the time the product team does not have the language resources needed to translate the project to a targeted language. An external localization vendor is a good solution for this problem. It is important to provide localizers with the needed components to do their job. A localization kit is a subset of tools, source files, binary files, and other localization information that is handed off to localization vendors. The localization kit should contain everything that localizers need to finish their job correctly, such as the localizable file types and names from the original-language product, the target localization languages, the localization instructions, schedule details, translation style and glossaries, procedures for creating binaries using the localized resources, and contacts on the product team. (For more information on localization kits and on making translation easier for localizers, see Chapter 2.)

By setting up communication channels between localizers and the appropriate people on the product team, you can speed up the localization process. Clear schedules of localization milestones make sure that the project will be delivered on time with high quality. The product team needs to make sure that their understanding of the overall internationalization process and of the localization process matches that of the localization team. For example, ensure that the text encoding used in localization is supported by the program. Localizers should be made aware of issues such as these from the outset.

The terminology that is going to be used in the translation process is based on localization glossaries; these glossaries should be reviewed before the translation process starts to make sure that the terminology suits the project objec tives. Terminology within software games will differ dramatically from terminology used within financial spreadsheets, so glossaries should reflect these distinctions.

Keeping track of the words that are actually translated for each handoff is important in order to make sure that vendors are paid for the actual work done. The localized product can also be tested by a third party, preferably by native speakers of the target language. Finally, the localization manager needs to make sure that there is a good archiving system of all files that are sent to and received from vendors.

So far this chapter has dealt mainly with software localization, in which changes are made to a program's resources. The final section of this chapter examines the second component of localization: content localization. Like software localization, content localization presents its own challenges. However, there are some notable differences between the two.


Content Localization

In addition to software localization, content localization is an integral part of adapting a product for a particular market. Content localization includes UA documentation, which consists of printed documentation, online content, and Help files. Content can be actual pieces of the software product or an online product by itself.

Software localization involves different methods than those used for content localization. One of the most obvious differences between localizing content and localizing software is that the Help text often consists of long continuous text that can be translated as one big piece. Even in the case of an online Help system, there are usually short topics that are connected to each other by hyperlinks. Each topic page still represents a complete entity in itself that is generally comprehensible to both the reader and to the localizer. Thus this text can be translated from the first to the last line, sentence after sentence, without the localizer ever having to wonder where a particular string might show up in the product or what it could mean. This is especially true since this text is designed to serve as documentation and is, by nature, somewhat self-explanatory.

Tracking content changes made in the source is crucial to lowering localization costs. Instead of having to retranslate the whole document, it is much more efficient to just retranslate those parts of the localized text that have been modified in the source. Automatic translation tools, which are common for content translation, use a feature called "Translation Memory" to help track content changes. That is why the content should be written in a way that is compatible with these tools in order to reduce translation errors and proofreading efforts. The content's text needs to be isolated from the layout and structure tags. Isolating these tags reduces the possibilities of inadvertently mistranslating something. The layout needs to be designed in a way that is compatible with the languages into which you are localizing, so that you don't have to make many modifications to tags. Such general layout design makes the localization job proceed more easily and quickly.

One of the big challenges in localizing online Help and related Web sites is that the original text is often created under a very tight schedule toward the end of the product cycle. If the localized product is scheduled to ship simultaneously with the original-language product, the documentation has to be localized under an even tighter schedule. Indeed, adding the localized Help to the build is often one of the last things to be done before a localized product is shipped, which does not leave much room for testing. And while there is little chance of the localized Help breaking functionality within the product itself, you still want to be sure that all the links that connect the different Help pages still work.


Summary

Localization of a product requires that the product be adapted to both the language and the culture of a particular market. Unlike a translated book, the product must adhere to a particular locale's conventions and culture. In fact, the final localized version of the original product should look and feel as if it had been designed in the user's home country. Avoiding sensitive geopolitical issues is one important consideration in making sure a product is well received in the international market. Some examples of other things that need to be adapted based on locale include currency, address, number, and date formats; the sort order; and for many Asian languages, the fonts and the font size. RTL languages such as Arabic and Hebrew require layout not only of the text, but of the whole UI, including buttons, menus, and dialog boxes. There are many UI elements that are modified in both software and content localization. These include text, layout, graphics and multimedia, keyboard shortcuts, fonts, character sets and locale data, as well as the product's build process and packaging.

The features of a good localization tool include the ability to change dialog font names, sizes, and styles; to resize, move, and hide controls as necessary; to change the encoding of the text; to replace graphics and icons with localized ones; to modify keyboard shortcuts; and to extract and edit different resource-file formats. In addition to obtaining capable tools, you will need the right assortment of individuals to carry out a successful localization project. A large localization project requires a team consisting of a program manager, one or more localization engineers, and many localizers. The localizers should possess not only excellent language skills but sound technical knowledge also.

Establish clear guidelines for ensuring effective localization, for scheduling international releases, and for localization outsourcing. The initial product speci fications should consider localization as part of the product cycle. Delaying localization to the end of the cycle can delay the shipping date because of any unexpected localization problems that require code changes. The product team and the international marketing team should decide what the delta between the shipping dates for the original product and the localized product should be. If you are outsourcing localization, provide vendors with a localization kit.

Content localization involves printed documentation, online content, and Help files. In contrast with software localization, content localization can usually be translated as one big piece, without the localizer ever having to wonder about strings or what their meaning is. The content should be written in a way that is compatible with localization tools. Additionally, the content's text needs to be isolated from the layout and structure tags. A localization strategy that is well planned and executed will help you produce a successful localization project.


Part V: Testing

When implementing a world-ready design and development process, all changes and decisions you make from beginning to end will need to fit into the framework of world-readiness. Since testing will also need to adhere to these concepts, Part V, "Testing," focuses on issues fundamental to a world-ready approach to testing. You might think ascertaining whether applications are world-ready merely entails testing localized applications. Instead, testing must verify that world-ready criteria have been met throughout the entire design and development process, from the earliest stages onward. Chapter 11, "Testing for World-Readiness," discusses the importance of globalizing the test process, while also looking at the problems associated with test processes that aren't world-ready. The chapter helps you address these issues by providing practical solutions, such as by showing you how to globalize the test process. You'll also see how to detect specific areas of functionality that might have globalization problems, and learn how best to conduct localizability testing and localized testing.

At the end of Chapter 11, you'll find "Sample International Test Cases." This list (though not all-inclusive) breaks up test items into basic areas of functionality. It shows you how to verify these areas of functionality, tells what the potential problems associated with certain areas are, and pinpoints the circumstances in which you would conduct a particular test.

Chapter 12, "Testing Localizability with Pseudo-Localization," expands on the role of pseudo-localization within localizability testing, introduced in Chapter 11. In addition, the chapter gives examples of common localizability bugs and lists some of the features associated with pseudo-localization-including translation of text resources and text-length extension. Also discussed are pseudo-localization of graphics and audio, and of Help files and Web content. Finally, the chapter examines pseudo-mirroring for bidirectional (BiDi) languages.


Chapter 11: Testing for World-Readiness

From previous chapters, you know that creating a world-ready application is a possible task and a desired goal. However, as you've seen, world-readiness is not guaranteed just because the target platform or the technologies used are world-ready-the whole design and development process must be globalized to achieve world-readiness. This need for globalization extends to software quality assurance (QA) processes as well.

Though the primary thrust of this chapter is on testing, the scope of the QA organization extends beyond this. Included under its umbrella is a wide array of tasks, such as reviewing architectural design and code, creating review processes, defining practices that will improve the development process, and even gathering marketing information. Generally, employees scattered throughout the company perform most of these QA tasks, rather than a dedicated team (as is usually the case with testing on larger projects). The individual processes that make up QA differ from company to company; consequently, this chapter cannot readily address or account for all the varying procedures in place within companies. However, the one common denominator among the QA organizations within these companies involves the testing process, which is generally handled in a fairly similar manner. For this reason, this chapter focuses primarily on making the testing aspect within the QA organizations world-ready.

While QA in all its varying facets must be globalized if you want to ensure world-readiness, legacy testing processes will require the most adjustments to achieve this very attainable goal. The nature of these changes basically involves adding another layer to the test process that was established without globalization in mind-namely, requiring that all design documents, practices, and procedures be world-ready. It is not always an easy task, as implications and requirements of going global might not be obvious in the development process, but it is essential given the limitations of legacy testing processes. (See "The World-Ready Approach to Testing" later in this chapter.)

This chapter will help you organize and implement your test process to further the development of a world-ready product. First and foremost, you'll need to globalize your test and, only if you plan to localize, conduct localizability testing and localization testing. To lay the proper foundation for carrying out a globalized test, the chapter will first help you understand the problems with legacy testing methods in terms of having a QA organization that is world-ready. It will look at the far-reaching benefits associated with globalizing existing tests and point out general areas of functionality that might have globalization problems. (See the "Sample International Test Cases" at the end of this chapter for further areas of functionality that you can test.) Technical solutions (such as how to actually detect and recognize specific globalization problems) follow this foundation. Finally, you'll see the purpose and scope of localizability testing and localization testing, ways to perform them, and the role of tools in today's test world.


Quality Assurance for World-Readiness

The goal of ensuring world-readiness has a wider scope and requires more fore-thought than merely testing localized applications. It must start early on by determining the necessary criteria for globalization and hence world-readiness. Then the QA organization must make sure that those implicit requirements are met throughout the design and development steps.

For software products shipped to multiple markets, the single, world-ready binary dramatically simplifies the QA process. Indeed, before software globalization, the development process entailed concurrent development of essentially different products-language-specific binaries-to ensure they all had matching functionality. By using a single, world-ready binary, however, you only have to deal with one product built for one specification, one development team, and one binary that can be tested, tracked, and supported later on. In addition, expanding the target market of the product becomes easier and more efficient, since you do not need to retest the single, world-ready binary completely in order to sell it in another country.

You can achieve all these benefits if the application you build targets a world-ready platform, like Microsoft Windows 2000 or Microsoft Windows XP. However, just because a platform is world-ready does not necessarily mean that an application's code is automatically globalized. Software design and development can easily break world-readiness when:

  • Legacy, single-language methods and practices are used. (See Chapter 2, "Designing a World-Ready Program." )
  • New components interact with components built in the pre-Unicode era. (See Chapter 3, "Unicode." )

Making software world-ready is a new goal for many software companies, and problems are likely to arise when new technologies and methods are applied. The QA organization, especially those responsible for testing, must recognize the problem with test processes that are not world-ready-as shown in the next section-and must be prepared to prevent those problems or reduce their negative effect through proven solutions. (See "Globalization of the Test" later in this chapter.)


The World-Ready Approach to Testing

Glossary
  • Enabling: Altering program code to handle input, display, and editing of bidirectional or East Asian languages, such as Arabic and Japanese, respectively.

Single code-base design, unified data-processing algorithms, and elimination of functionality problems due to localization are some of the benefits of software globalization. These benefits, which were not possible with former development methods, reduce redundancy in test runs on localized products and make simultaneous release possible.

The old approach to international testing was extensive and often inefficient, as was international software development. If you wanted to double the number of supported languages or locales, you could end up doubling the resources spent on development and testing. Even if the settings for the two locales were similar, you could not assume the application would work for one locale just because it worked for another.

By globalizing the functionality test, however, you can make testing easier and more efficient. Once you know the application is globalized, you know it supports any locale and any language. Unlike older test processes, the world-ready approach renders unnecessary the software development concept called "enabling." When done correctly, all world-ready applications are enabled to handle all the languages that are supported by the underlying platform. Thus instead of having to test functionality for each localized version of an application, by globalizing the functionality test you verify that functionality has been globalized for the core product. Even though global functionality is available in a single, world-ready binary, localized products are still in demand in their respective markets, and thus still need to be built and tested. What is different is the focus of localization testing. Now you only need to check how well the product was adapted for a given locale or language, which allows you to verify that localization does not break globalized and localizable resources. Since this adaptation mostly involves translation, localization testing requires only limited technical expertise; instead, language proficiency becomes the major requirement for the tester. (See "Localization Testing" later in this chapter.)

In addition to localization testing becoming easier, the entire localization process in general becomes less complex and requires fewer resources. This is yet another of the many advantages that software developers gain by creating a single, world-ready binary. (See Chapter 2.) The following list is just a few examples of how the localization process becomes more efficient:

  • Localization teams are less likely to find localizability problems in multiple language versions that might require multiple fixes and branches of source code.
  • Since the localization process becomes mostly user interface (UI) translation, localized product testing becomes less technical and more a matter of checking grammar and spelling.

But before localizers can start their work for the very first target language, those responsible for testing must verify that the application is globalized and localizable (world-ready). Figure 11-1 shows the steps needed-from a testing perspective-to ship a world-ready and localized (internationalized) product. While the figure necessarily omits the many details of a world-ready development process, and while the actual sequence of tasks might differ sometimes, it assembles the major components of this chapter in their logical order. Most importantly, the figure reinforces some key ideas with regard to globalizing the QA process:

  • Multilingual sim-ship of software is possible if core development and testing meet world-ready criteria.
  • Should you decide to localize, the localizability test should be performed in parallel with the globalized functionality test.
  • Localization must start as soon as the code is stable enough, and only after the initial localizability test.
Figure 11.1 Testing process required for shipping an internationalized product.
Figure 11.1 - Testing process required for shipping an internationalized product.

Another key implication of globalization-and globalized testing-is the ability to expand the target market after the product is shipped. Thus even if supporting some particular locale or language was not on your mind when the application was developed, you can re-evaluate later and move to a new market without changing the code. Globalized testing performed in the development cycle will ensure there are no serious problems with this move. Of course, some basic language-specific or locale-specific verification is required, but the amount can be significantly reduced compared to what was needed before releasing a language-specific application. When properly conducted, globalized testing will decrease the number of bugs, help you use resources more efficiently, and reduce costs down the road. The next section will show you how to prepare and carry out an effective globalized test that guarantees the globalization of the application, and will also discuss some common globalization problems.


Globalization of the Test

The first step to test the world-readiness of a product is to verify that its functionality is globalized. As you've seen, this is done by modifying legacy test cases-or globalizing the test. Globalization represents functionality that is an intrinsic, inseparable part of a system or product. Rather than testing for globalization of every single system component of multiple products, which would require far too many resources and hours, by globalizing the test you check for world-readiness at the same time you check the general functionality of a core product's entire system.

Although Part II, "Globalization," explored the concept of globalization in depth, in order to help you verify that a product's functionality has been globalized, some central ideas of globalization are restated here. Among these ideas, an application or a component is considered globalized if:

  • It is Unicode-based; that is, any Unicode text can be handled successfully.
  • Correct encodings are selected for text conversions, if conversion is needed to interact with legacy systems.
  • The code follows the user's culture-specific settings.

Whenever a piece of code handles text or deals with locale-sensitive functionality, it has to be tested for proper functionality from the perspective of world-readiness. Code that has been globalized can process data (text) in multiple scripts and can adjust itself to the environment (in terms of locale settings, for instance). Verifying functionality can be planned as a separate test pass, but it is better to fit it into regular functional testing. (For a detailed description of steps required for code globalization, see Part II.)

Practical guidelines for globalization of your test go beyond breaking the code-page dependency in the test data and checking the cultural accuracy (which means matching the user-locale settings). To make testing more effective in searching for globalization bugs, target specific functionality and create an environment where this functionality is likely to break. Here are some of the areas of functionality that testers should verify:

  • Multilingual text processing with no data loss. Unicode data that does not match locale settings should never be converted to single-language encodings, hence preserving all data.
  • Correct encoding conversions. When the data, for certain design reasons, needs to be converted to or from nonmultilingual encoding, correct encoding should be selected when multiple choices are possible. These choices include Original Equipment Manufacturer (OEM) code pages versus Windows code pages, UTF-8 encoding versus OEM encoding, and so on. For double-byte text handling, when source or target encoding is multibyte, bugs in code that handle text in multibyte encodings should be visible even in an otherwise globalized application. Also, make sure that all conversions to non-Unicode encodings are justified-do not let incorrect assumptions about encoding limitations corrupt your text data.
  • Proper handling of locale settings. The system locale should affect only encoding conversions. User-locale settings should define the date, time, number and currency formatting, calendar and sorting settings, and nothing else. Input locales should be properly handled, such that the input locale can be selected for any supported language, and this language can be entered.

(See "Creating the Test Environment" later in this chapter for specific ways to target and detect the problems just listed.)

You've seen the importance of globalizing your test as well as particular areas of functionality in which to search for globalization bugs. How do you now go about preparing a globalized test?

Preparing the Test

Before testing is performed, there are several items to consider in preparation. For example, you'll need to determine priority levels for components that are going to be tested, choose a test platform, and create a proper testing environment.

Prioritizing the Components

Some components are more likely to have globalization problems than others, so part of globalizing the test means adapting your testing practices accordingly. Assign a globalization priority to all components that are going to be tested. Top priority goes to:

  • Code designed to be used on Windows 2000 and Windows XP, and on Microsoft Windows 95, Microsoft Windows 98, and Microsoft Windows Millennium Edition (Me). Top priority also goes to code that interfaces with components written for Windows 95/98/Me or earlier platforms.
  • Components that handle text data in non-Unicode encodings. Some of them can be identified as those calling non-Unicode functions of the Microsoft Win32 application programming interface (API). Network components or components with console output are assumed to be in this area. Components using standard C run-time (CRT) libraries to handle text input/output (I/O) belong to this group, too.
  • Components that extensively handle text data.
  • Applications using files for data storage or data exchange (Windows metafiles, Microsoft Jet database files, Group Policy Engine, security configuration tools, and so on).
  • Components that have had many globalization problems in the past.

After determining which components are more likely to have serious globalization errors, choose a viable test platform in accordance with your particular circumstances.

Choosing a Test Platform

Choosing the operating system for your globalized test is another important step. Among the factors that can affect your decision are the functionality of your system and required level of support, testers' ability to work on a localized version of the operating system, and availability of the operating system for testing.

Even if you are targeting a broader range of operating systems, Windows XP can be a good fit for your primary test platform. No other operating system gives you the same flexibility with locale settings, in addition to native support for the broadest range of languages and locales. Windows 95 and Window 98, for example, keep a tight rein on your ability to change the system locale. Windows XP, on the other hand, gives you freedom to adjust system and user locales. In fact, by adjusting these settings, even when you run a localized application on an English version of Windows XP, the application will still behave as though it is running on a localized platform. Thus you can simulate varying language and regional environments on one machine, according to which system and user locales you set. This would normally only be possible by having multiple computers running different language versions of the operating system.

You can also use other platforms that differ from English Windows XP. For instance, Windows XP Multilingual User Interface Pack, henceforth referred to as MUI, is especially useful if your code implements a multilingual UI and has to adjust to the UI settings of the operating system. This approach is a more easily implemented alternative to installing multiple localized versions of the operating system. (For more information on MUI, see Chapter 6, "Multilingual User Interface [MUI].")

Another test platform you can use is the localized build of the target operating system. If the tested application is to be localized into certain languages, these languages are the obvious choice for the platform. If the target operating system is Windows 2000 or Windows XP, other language versions of the operating system can be used too, even if the UI language does not match the language of your application. By using this configuration, you check how your application interacts with a localized system, where names of the system folders, built-in accounts, fonts, and other system objects might be different from what you get with an English or MUI system. Once you've chosen a test platform, the next step is to create the test environment.

Creating the Test Environment

Choosing a platform for the test is not enough to guarantee that the test is globalized and that no globalization problems are left in the code. The static properties of the platform (such as names of the system folders, built-in account names, and so on) are only one part of the picture; you'll also need to consider those properties that can be manipulated-in other words, the platform's dynamic properties. For instance, you can configure language and cultural preferences of Windows through different settings of system and user locales. You should, in fact, tune these settings for the best outcome of your test. While this chapter has identified general areas of specific functionality to target, you have not yet seen how to target them. Therefore, the following list digs deeper into those areas, giving you optimal test settings to use for detecting some typical globalization problems. (For more information, see the "Sample International Testing Cases" at the end of this chapter. This checklist gives you further areas of functionality that you can test, along with showing you how they can be verified, the potential problems associated with them, and when the test is applicable.)

Non-Unicode data path cannot handle double-byte data properly. The problems in this area can be detected if you select an East Asian system locale such as Japanese, Chinese, or Korean, and use data encoded with double-byte characters in the corresponding code page.

Components interacting with non-Unicode applications select incorrect code page (Windows versus OEM) for character conversion. This problem can be detected with the system locale in cases where Windows and OEM encodings do not coincide. European locales, like German or Russian, are good candidates. The test data must contain characters whose code-point values are different in Windows and OEM code pages.

The component is unable to process and follow user-locale settings. Select the locales where special rules must be applied to functionality that your application exposes. For instance, for special sorting rules, select Turkish (test with dotless "I"). For details, see "Globalizing Your Test Data" later in this chapter. Select Chinese for applications with different sort orders (and use Chinese data).

For calendar processing, select locales with non-Gregorian calendars. Finally, for date, time, number, and currency formatting, select locales with formatting rules different from those accepted in the environment where the application is developed. Try locales with different time and date separators, since too often programmers assume the colon or slash to be the only possibilities. For example, the Italian (Italy) locale uses a period (.) as a time separator, while the Italian (Switzerland) locale uses a period as a date separator. In addition, make sure that data (such as files and databases) saved under one locale setting can be properly read and used under another locale, and that the data shown matches the client's settings.

It is also a good idea to set the user locale different from the system locale. A non-Unicode component will fail to display when all of the following are true:

  • Your application picks an incorrect locale for data formatting.
  • The mechanisms used to display locale-specific data treat text as Unicode.
  • The date or time has to be displayed using symbols not in the current code page.

Having a non-Unicode component that fails to display is as bad as picking an incorrect format.

For distributed applications, data is not successfully passed between different locales. To detect this problem, set up a network with a mixed environment where some systems have an East Asian (for example, Japanese) system locale and others have a European system locale. Having the application run in a distributed, mixed network verifies that data can be successfully passed between different locales. Additionally (though not really within the scope of globalization), it might also be wise to test how your distributed system works when the time-zone settings are different among computers in your system.

You've determined which code, components, and applications are most likely to present globalization problems, decided upon a test platform, and manipulated settings to create an optimal testing environment for detecting specific problems. Following the guidelines just presented will ensure you have considered vital issues when preparing to carry out a globalized test.

Carrying Out the Test

In order to carry out a globalized test, you will need to globalize your test data, making sure your test covers functionality-specific areas. You will also need to be able to recognize globalization problems such as loss of functionality, as well as code that fails to follow locale conventions as defined by the current user locale.

Globalizing Your Test Data

After the environment has been set for globalized testing, your regular test cases must be run with special attention paid to potential globalization problems. The test data must match the test. Generally, it must be multilingual Unicode text that doesn't match the system locale. Use a mix of scripts (Latin, Cyrillic, Thai, and so on) or Unicode-only languages (Armenian, Georgian, Indic languages) to discover code-page dependency.

Some particular areas of functionality might require special test data. For example, if the application converts the case of letters, you can verify the functionality with scripts that do not have uppercase and lowercase symbols, such as East Asian scripts or Hebrew. You can also use text with special locale-specific rules for handling case conversion, such as Turkish text with "dotless I." In Turkish both the "dotted" and "dotless" letter "I" exist. "Dotless I" converted to lowercase becomes " [image] ", while "dotted [image] " converted to lowercase becomes "i".

Another example of an area requiring functionality-specific test data is the sorting procedure, for which there are some specific examples of data you can use. First, Latin text must be sorted differently based on the differences between the Modern Spanish and Traditional Spanish sort order. The letter combination "ch" is considered a unique compression under Traditional Spanish sort order. However, under the Modern Spanish sort order, "ch" is considered as just the separate letter "c" followed by "h". Traditional Spanish sort order also defines the combination "ll" as a unique letter, while Modern Spanish sort order considers "ll" to be just the letter "l" followed by another letter "l".

A further example of test data involves the sorting order for Turkish "dotless I" versus "dotted I." Under Turkish locale settings, "dotless I" (the uppercase "I" or the lowercase " [image] ") precedes "dotted I" (the uppercase " [image] " or lowercase "i"). In other words, in Turkish sort order "I" precedes " [image] ", and " [image] " precedes "i".

Some characters in your application have the potential to cause functionality problems. If the application has special handling rules for specific values of characters, some characters incorporated into the data stream will be processed as special control codes. Characters whose code points contain byte sequences that coincide with these special control codes can cause functionality problems.

For example, suppose an application has to parse a string that presumably contains a folder name. The algorithm will have to search for a backslash (the standard delimiter, or path separator, that Windows uses to denote folders). If the algorithm is designed to search for the byte value of the delimiter, it will have to stop at symbol 0x5C, the byte representation for the backslash. However, 0x5C is a valid trail byte in East Asian code pages, so the algorithm will fail on certain strings containing those characters. This will not happen if the algorithm is designed to search for the backslash character itself instead of its byte value-namely, if characters located with the CharNext( ) function are examined for matching the criteria, not the values of bytes in the string.

When planning your test, put greater importance on test cases that deal with the direct or indirect I/O operations on strings. You should also give special priority to tests involving text and formatted data (such as numbers, monetary values, date, and time). Direct input happens when a user enters text in the UI or the text is fetched from some external storage, like a text file or database. Examples of indirect string input are host names, user names, and folder names that include current user names, and so on. If your application deals with information of this kind, make sure this data is multilingual when your tests are running.

In a small number of cases the range of characters in the test input might be somewhat limited, in accordance with the design limitations. For example, only characters that match the system locale can be used in NetBIOS-related tests. Set these tests in a globalized environment. (See "Creating the Test Environment" earlier in this chapter.) It might be hard to enter all of these test inputs manually if you do not know the languages in which you are preparing your test data. A simple Unicode text generator can be very helpful at this point. (You can find a sample of this utility in the Samples subdirectory of the companion CD.)

In addition to globalizing test data that will catch problems and validate functionality, as you conduct your test and view the results you should be able to recognize how some of the globalization problems mentioned earlier in "Creating the Test Environment" manifest themselves.

Recognizing the Problems

The most serious globalization problem is functionality loss. This problem can surface either immediately, such as when a system locale is changed, or later, such as when accessing input data (non-Latin character input).

Some functionality problems manifest themselves in the form of display problems. The following are some common ones that you might see:

  • Question marks (????) appearing instead of displayed text indicates problems in Unicode-to-code-page conversion.
  • Random characters ( [image] and so on) appearing instead of readable text indicates that the code page-based code is using the wrong code page.
  • The appearance of default glyphs such as boxes, vertical bars, or tildes ( [image]) indicates that the selected font cannot display some of the characters.

It might be hard to find certain problems in display or print results without having adequate shaping, layout, or script knowledge. Testing for these sorts of problems is language-specific and often cannot be executed without language expertise. On the other hand, this type of test might be limited to code inspection. For instance, if the text output is formed and displayed using standard text-handling mechanisms, like Uniscribe for complex scripts (see Chapter 24, "Uniscribe"), complex-script processing in text output can be considered safe.

Another area of potential problems is code failing to follow locale conventions as defined by the current user locale. Make sure that locale-sensitive data in your application (numbers, dates, time, currency, and calendars) is displayed according to the current settings in the Regional And Language Options property sheet of your computer. (For information on locale awareness, see Chapter 4, "Locale and Cultural Awareness." )

However, Regional And Language Options does not cover all locale-specific functionality. For example, you cannot see the current sort order there. Thus it is important to have a test plan covering all aspects of functionality related to locale before you start your test. You can use the National Language Support (NLS) and the .NET Framework documentation as a starting point for this plan. Use this documentation for finding exactly what locale information you'll need to retrieve dynamically, and then apply those requirements to your project.

As a final note regarding globalized testing, when planning and running testing always remember that globalization is wider in scope than just supporting one particular language. The test you use to verify that code has been globalized might not seem to duplicate a realistic usage scenario; that is, the settings of a particular globalized test may look a bit unnatural. Think about cars being tested in extreme weather conditions. Obviously the deserts of Arizona in midsummer or the frozen Alaskan tundra in midwinter don't represent typical driving conditions. However, by covering the widest range of conditions possible, the test will produce more comprehensive results. Similarly, your test should cover the widest range of possibilities you can think of#151;including potential markets you might never have considered or even heard of, as well as the multiple ways in which your product might be used.

Once you've verified that a product's functionality has been globalized through globalizing the existing testing process, you will need to determine whether the application is going to be localized. If so, the next step is localizability testing. Using techniques such as pseudo-localization and the pseudo-mirroring test; reviewing code, UI, and documentation; and performing pilot localization are all tasks that comprise localizability testing.


Localizability Testing

The purpose of localizability testing is to verify that the UI of the program being tested can be easily translated to any target language without re-engineering or making code modifications. Since localizability bugs have to be fixed in the code of the application, they must be looked for at the earliest stage of development possible. By doing so, you can circumvent having to fix bugs for every language version at a later point. (See Chapter 2.) Paradoxically, however, the problems that cause bugs to appear in the first place usually only become apparent once localization is done. Furthermore, since localization occurs only after the code is complete and reaches a certain degree of stability, there is inevitably a long time span between an application's development and its localization. There are four things you can do to resolve this contradiction.

  1. Run a pseudo-localized version of a program. Pseudo-localization can be the most effective way of finding localizability bugs, which are generally detected when translating the program's UI. Pseudo-localization gives you a translation without the cost of an actual localization. (For information on how to create a pseudo-localized version of your application, see Chapter 12, "Testing Localizability with Pseudo-Localization.")

    Once you pseudo-localize your program, test its functionality. Pseudo-localized applications should function no differently than their original version. Besides detecting localizability bugs, pseudo-localization often reveals globalization problems as well. As a result, testing a pseudo-localized application can potentially be central to verifying that an application is world-ready.

    An area often forgotten in localizability testing is the pseudo-mirroring test, which can be implemented as part of the pseudo-localization of your product. If you want to distribute your software to markets where the text and UI of the programs display from right to left, you should follow the development and design rules outlined in Chapter 8, "Mirroring," to allow mirroring of the UI. Pseudo-mirroring then verifies that these rules were applied. Testers do not need to wait for the localizers to mirror the UI of the application-pseudo-mirroring can be applied to the code. When conducting the pseudo-mirroring test, you can allow the UI text to remain unchanged and to be displayed as usual; only the windows and text alignment will be mirrored. (For more information on pseudo-mirroring, see Chapter 12.)

  2. Perform code review. When reviewing your code, be sure it meets the following requirements:
    • All resources are separated from the code and are written in standard resources format. This does not mean you have to use Win32 resources only; the goal is to have them isolated from the code, easy to maintain and localize, and kept in a form that allows the application to access the proper language version of the resources.
    • Pointer arithmetic is not used for string-length calculations, access to string elements, or string manipulations. All text manipulations and parsing-such as finding string length, locating substrings, changing text case, and so on-have to be done using locale-sensitive and encoding-sensitive library routines. (See Chapter 5, "Text Input, Output, and Display.")
    • Strings are not built at run time by stripping or concatenation. String substitution is done via language-aware APIs such as FormatMessage.
    • Resource-handling mechanisms do not make assumptions about string buffer length. For example, if the application defines a name of a user with administrator's privileges and stores it as a string resource, it won't be a good idea to load it from a resource into a fixed buffer of 14 characters, just because the null-terminated string "Administrator" takes 14 characters. Since translation is likely to increase this length, the localized name might not fit into the buffer.
    • UI controls are not positioned at run time.
    • Icons and bitmaps do not contain text. Generally, try to make your graphics locale-neutral. Or, if they have to be locale-specific, the comment "localization required" must be attached and passed to the localizers.
    • No assumptions exist on drive and folder names or registry keys. The names of the system folders can be translated; the same can apply to the names of other objects defined by the operating system and applications your code interacts with. Folder names, names of built-in accounts, and other localizable elements of the operating system can and should be queried at run time. If your application defines localizable objects of this kind, make sure they can be changed with no modifications to the code. For example, if an installation routine creates folders with localizable names, make sure that code working with those folders can find them using the same localizable resource or some language-neutral mechanism like an environment variable.
  3. Perform UI and documentation review. Make sure the terminology used in the UI and support documentation is clear, consistent, and unambiguous. Localizers find it hard to do their jobs when the UI and the documentation refer to the same features but use different words, or when the text is overloaded with technical slang. Content review becomes even more important if your localizability relies on pseudo-localization. Pseudo-localization helps to find code problems; automatic machine methods cannot find language that is too technical, unclear, or inconsistent with that used in other parts of the documentation.
  4. Run a pilot localization project. Though there are many advantages to pseudo-localizing your application, you might opt to run an actual pilot localization project (such as if you are localizing for only a couple of European languages or if you cannot afford to build a pseudo-localized version of a product). If you choose this method, first, you should use one language for which you can easily organize localization and testing. For instance, maybe you have experience with this particular language from your previous projects, or your organization employs people who speak this language.

    5. Second, you should choose a language for which the localization is especially likely to expose localization problems. German can be a good example because the translated text is likely to grow in length. For example, "cut (and paste)" translates as "ausschneiden (und einfügen)" in German. Also, Windows encoding differs from OEM for German.

    Third, choose a language for which the target market of the localized product is of great importance to you. The pilot language version will be the first product that is ready for shipping.

    In spite of its benefits, pilot localization cannot guarantee that all issues will be covered in your localizability test. Different language versions might vary in the degree of translation required and in the problems you are likely to see due to localization. By using pseudo-localization to address general areas of potential risk, you will preempt more problems than by using pilot localization, which focuses more on the requirements dictated by one particular language.

As illustrated earlier in Figure 11-1, globalization and localizability testing are the two types of tests conducted within the development process of world-ready software. The final test, localization testing, technically lies outside the realm of world-readiness, but is still an integral part of creating an application that's internationalized-in other words, in creating an application that is both world-ready and localized.


Localization Testing

Although localization and, by extension, localization testing are not strictly a part of the development of world-ready software, localization becomes possible once you have developed world-ready software. If you do decide to localize, you should be familiar with the scope and purpose of localization testing. Localizers translate the product UI and sometimes change some initial settings to adapt the product to a particular local market. This definitely reduces the "world-readiness" of the application. That is, a globalized application whose UI and documentation are translated into a language spoken in one country will retain its functionality. However, the application will become less usable in the countries where that language is not spoken.

Localization testing checks how well the build has been translated into a particular target language. This test is based on the results of globalized testing where the functional support for that particular locale has already been verified. If the product is not globalized enough to support a given language, you probably will not try to localize it into that language in the first place!

You should be aware that pseudo-localization, which was discussed earlier, does not completely eliminate the need for functionality testing of a localized application. When you test for localizability before you localize, the chances of having serious functional problems due to localization are slim. However, you still have to check that the application you're shipping to a particular market really works. Now you can do it in less time and with fewer resources. The following section shows you some of the general areas on which to focus when performing a localization test. (For information on more specific areas, see "Localization Testing of the UI" later in this chapter.)

General Areas of Focus in Localization Testing

Localization testing should focus on several general areas. The first involves things that are often altered during localization, such as the UI and content files. The second consists of culture-specific, language-specific, and country-specific areas. Examples include configurable components-such as region defaults and the default language-as well as language-specific and region-specific functionality-such as default spelling checkers, speech engines, and so on. You should also test the availability of drivers for local hardware and look for the encryption algorithms incorporated into the application. The rules and regulations for distribution of cryptographic software differ from country to country.

Pay specific attention to the customization that could not be automated through the globalization services infrastructure (Win32 NLS APIs and the .NET Framework). For example, check that formatting of mailing addresses is locale-specific and that parts of the user's name are ordered correctly. (The order in which surname and first name appear varies according to country. For instance, some Muslim countries and certain regions in India use a different name order than that used in the English language.) Functionality of this kind is often implemented by an application-testing must verify its correctness.

Other areas of localization testing should include basic functionality tests; setup, upgrade, and uninstall tests that are run in the localized environment; and, finally, application and hardware compatibility tests that are planned according to the product's target market.

Platform in Localization Testing

Any language version of Windows XP or Windows 2000 can be selected as a platform for the test if the product is properly globalized. Of course, in the case of localization testing, the localized version of the operating system can be a wise choice, since that's the most likely environment for your application in the real world. However, a globalized and localizable application, even after it undergoes localization, must be able to run on any language version of the operating system and with MUI installed.

You should run the application with MUI installed when your application implements an MUI behavior, through pluggable UI, satellite dynamic-link libraries (DLLs), or some other technique that adjusts the UI language to the user's preferences. MUI allows the user to switch the UI language of the operating system and thus you must make sure your application matches the operating-system settings. You should verify the behavior of the application when the user's default language of the UI differs from the other locale settings. By doing so, you'll immediately see any problems in the way resources are loaded and processed.

Localization Testing of the UI

Also keep an eye on the behavior of applications that run processes in a system-such as operating-system services-rather than in a user's context. When a system process queries its user default UI language settings, it might get a result different from what a user's process running at the same time will get. This can cause localization problems, inconsistency in the UI that the user sees (if parts of it are generated by the system services), or even problems in functionality. In order to avoid those problems, always check an application's behavior with different default user and system UI languages. The settings for UI languages should also be different from those used in the development environment.

For example, assume you have a machine with MUI installed and a user whose default UI language is different from that of the system. Suppose a fax service waiting for incoming calls is running continuously and that, when a fax arrives, the service displays a notification message to the currently logged user (if there is one). You must ensure that the message be in the user's language, which might not necessarily be the same as the one returned to the fax service when it queries its default UI language.

In particular, localization testing of the UI and linguistics should cover items such as:

  • Validation of all application resources.
  • Verification of linguistic accuracy and resource attributes.
  • Checking for typographical errors.
  • Checking that printed documentation, online Help, messages, interface resources, and command-key sequences are consistent with each other. If you have shipped localized versions of your product before, make sure that the translation is consistent with the earlier released versions.
  • Confirmation of adherence to system, input, and display environment standards.
  • Checking usability of the UI.
  • Assessment of cultural appropriateness.
  • Checking for politically sensitive content.
  • Making sure the market-specific information about your company, such as contact information or local product-support phone numbers, is updated.

It's also a good idea to check that everything you are going to distribute in a local market complies with the local laws and regulations. (See Chapter 2.) This includes not only the license agreement but also online Help and other user documentation.

You've now seen the importance of globalized testing, localizability testing, and localization testing in terms of ensuring both that a product is world-ready and that it is suitable for worldwide distribution. One way in which the testing process has changed to reflect new goals involves the type of tools currently being used.


The Role of Test Tools

Test tools play an essential role in today's testing process, particularly those that are automated. While automated testing cannot replace manual testing completely, many test areas gain tremendous benefits from its introduction. For instance, using automated test tools is an effective way to verify the functionality of localized products or to test the degree of a product's globalization.

By running automated tests on the original and localized applications, side by side, you can verify that localization of the application does not break the functionality. Furthermore, with automation you do not need to understand the language of the localized UI since the tools can test the functionality regardless of the UI language. (For instance, "Click the default button"-where the word "default" in this case is used in the generic sense-is a test of functionality that can work for any language.) In addition, ensuring that a product is globalized requires the ranges of test input to be extended and environment settings to be more diverse during the test than was the case before creating world-ready software. Automation makes it easier to deal with this increase in the volume of test cases and allows you to efficiently track the results of the globalized test. Restoring the locale settings or test input that brought about a failure in a program also becomes easier.

Automation of test runs, however, involves more than just running old, trusty test tools on your new product under a globalized environment. Test tools that check functionality of an application's UI can be broken when the tested application is translated. Even when not affected by translation, test results may be incorrect if the tool verifies dates, for example, and assumes that the date format is fixed. (Of course, in globalized applications, the date format varies according to locale and region. See Chapter 4.) The inability to use international test data can also make a test tool unusable.

In order to avoid these and similar problems, developers of test tools must follow the same rules as the developers of globalized software do. Test tools must be globalized, must adjust themselves dynamically to locale settings, and should be able to process international text data. Also, if they have to access localizable objects by name, test tools must be localizable (which in this case means they need to allow for easy translation of UI elements).

At Microsoft, test tools and techniques vary from one team to the next. Experience shows that some tools are easier to globalize, while others are not. More advanced programming models usually provide better ways to make the code universal in regard to locale and language settings.

Test tools written in C++ to test the operating system APIs rarely are dependent upon locale; if they are, those dependencies can be easily broken. Scripts accessing the UI of the tested applications through Component Object Model (COM) Automation interfaces are rarely broken due to localization of the UI.

The programming systems that are known to be problematic in globalized testing are generally built on non-Unicode APIs and are thus unable to handle multilingual text. Or the systems might lack the ability to query locale information and to format the output accordingly. Finally, systems that access the UI elements by their names can also be problematic in terms of globalized testing.

Therefore, if you base your automation testing on an older tool with a few or all of the limitations just listed, some of the steps required for globalization of the test might not be possible with your automation. First, try to substitute this test with a manual test. Second, try to make the tool useful for multilingual testing by doing the following:

  • Refer to UI elements by their resource identifiers, ordinal numbers, or window classes, not by the caption. Captions are hard to track with multilingual development. Non-Unicode test tools cannot access captions if they are localized into a language that doesn't match the system locale.
  • Isolate text resources from the code in a way that's easy to localize. For example, they can be an include file defining text constants. As a result, localizers will be able to translate the external "resource file," and your tool will run with a localized UI. The code-page dependency might still hold, though. Using a localized resource file might not be of any help if the system locale does not match.


Summary

For a product to be world-ready, the whole design, development, and QA process must be globalized. Consequently, the QA organization will need to add another layer to test processes established without globalization in mind-namely, requiring that all design documents, practices, and procedures be world-ready. After ensuring the functionality of the core product through globalized testing, determine if the product is going to be localized. If so, localizability testing and localization testing will be needed.

Whenever a piece of code handles text or deals with locale-sensitive functionality, it has to be tested for proper functionality. Globalized testing must go beyond breaking the code-page dependency in the test data and checking for cultural accuracy. Some general areas to focus on in globalized testing include multilingual text processing with no data loss, correct encoding conversions, and proper handling of locale settings.

Localizability testing verifies that the UI of the program being tested can be easily translated to any target language without re-engineering or making code modifications. Since localizability bugs have to be fixed in the code of the application, they must be looked for at the earliest stage of development possible. However, the problems that cause bugs to appear in the first place usually only become apparent once localization is done. Because of the inevitable long time span between an application's development and its localization, run a pseudo-localized test; perform code, UI, and documentation review; and run a pilot localization project.

Finally, localization testing checks how well the build has been translated into a particular target language and is based on the results of globalized testing where the functional support for that particular locale has already been verified. This test should focus on things that are altered during localization, such as the UI and content files. It should also involve culture-specific, language-specific, and country-specific areas; setup, upgrade, and uninstall tests that are run in the localized environment; and finally, application and hardware compatibility tests. Globalized testing, localizability testing, and localization testing, once conducted, complete the testing process required for shipping an internationalized product.


Sample International Test Cases

The following list is provided as a possible foundation for internationalization of test plans. The test items are broken up into basic areas of functionality and are accompanied by details on how these areas can be verified, what the potential problems associated with certain areas are, and when the test is applicable.

The checklist shown in the following tables cannot possibly cover every situation that might arise; nor can it address all the numerous areas of functionality within individual applications. However, it does offer valuable guidelines for some of the more common areas of functionality that you should test.

Globalization: Deployment and General Functionality
Areas and Items to Test | Details | Applicability
Application setup under different language environments | Install on a platform where the UI language of the operating system does not match the language of the application; verify with different UI languages, system locales, and user locales that do not match. Verify the ability of the application to work with localized operating-system objects. European-language versions of Windows might be the best choice of a platform because they are deeply localized. If the application implements an MUI behavior, make sure its UI language follows the UI language of the operating system. | All applications; if the application implements MUI functionality, make sure the default UI language of the application follows the UI language of the operating system.
Application setup on an operating system with MUI installed and where user's default UI language is different from English | Verify the behavior in a multilingual environment where the UI language is defined by rules different from those of the older systems. Change the system and user locales to verify multi- lingual abilities of the UI. | All applications; applications that implement MUI behavior must undergo testing that is even more extensive.
General functionality with various system and user-locale settings | Verify that the code does not have unwanted locale dependencies. | All applications
Uninstalling the application with different locale settings | Make sure that system-locale and user-locale settings that were active when the application was installed are different from those set at the time when the application is uninstalled. | All applications
Multiplatform installation | Applications that need to be installable on multiple platforms should call code page-based code for installation on downlevel plat- forms (Windows 95 and Windows 98). Make sure the UI that is displayed is not compromised on platforms with nondefault language settings. Unicode-based applications must work properly with different system-locale settings. No change in the user-locale settings or the user's UI language preferences should cause functionality or display problems. | Applications that target downlevel platforms or communicate with legacy systems
Globalization: Text Handling
Areas and Items to Test | Details | Applicability
Interactive text input using different input locales | Enter text while switching between different keyboard layouts and Input Method Editors (IMEs). Verify the ability of the application to accept input from various input methods, regardless of other locale settings. Either check that multilingual input is enabled with all supported input methods or that it is blocked for all languages except supported ones. | Applications that allow for interactive text input
Clipboard operations with multilingual text | Verify that multilingual text can be copied to and from the clipboard. Make sure the use of the clipboard (while cutting and pasting) does not compromise the limitations placed on localized text. Examples of such limitations might be those imposed on applica- tions interacting with non- multilingual components or systems, or when the text being passed on must match the system locale. | Applications that allow clipboard operations
Device-independent multilingual output | All methods of output-such as displaying UI text, printing, storing in an enhanced metafile (EMF)-must preserve the multilingual features of the text. These methods should not introduce problems due to different text encoding, font, or settings used for other output methods. | Applications that perform text output
Font independence | Verify that the names of fonts the application uses are not hard-coded. Run the application on a localized operating system where font names are localized and where default shell fonts are different from those used in the English operating system. (Localized Japanese Windows is a good example.) Change the default font settings of the applications. | Applications that have text in the UI
Text handling in the UI | Check that multilingual text can be edited, entered, and deleted. Verify that word wrapping breaks text prop- erly, regardless of the lan- guage. Make sure characters can be changed to uppercase and lowercase according to the settings of the language and the locale. Verify alignment of the multilingual output. | All applications that perform special text formatting; applications that use cus- tom solutions, rather than standard Windows controls, for text editing.
Cursor movement and positioning | Verify that the cursor can be positioned correctly at the start and end of characters and text. Check cursor movement and text selection. Pay greater attention to wide ideographic characters (such as kanji characters) and bidirectional (BiDi) text, since these can cause cursor movement problems if not handled properly. | Applications that don't use standard Windows controls for text editing
Handling of character encoding-double byte character set (DBCS) | Set an East Asian system locale; use text encoded with single-byte and double-byte values in the selected code page. Incorporate into the test data characters that might be "risky" for your application. (For information on detecting characters that could be risky, see Appendix N, "Risky Characters.") | Applications that implement non-Unicode text handling
Handling of OEM and Windows encodings | Set the system locale where some character code-point values defined by Windows and OEM code pages are different. For example, the code-point value of "small sharp s" (ß) is 0xE1 in code page 850, set with the German system locale. However, Windows code page 1252, which is associated with this same German system locale, defines this letter as 0xDF. Use characters such as ß in the test data, and verify that they are never displayed, stored, or retrieved as some other character. | Applications that implement non-Unicode text handling; network and console applications
Handling of complex scripts | Multilingual applications must process complex scripts correctly-either by using standard Windows controls or calling special complex-script handling routines like those of Uniscribe. Check that BiDi text is properly reordered, that Arabic text is correctly shaped, and that word-breaking rules are applied to Thai text. | Applications that implement custom complex-script handling; basic verification is required for all applications sold in markets where com- plex scripts are used.
Handling buffer sizes for multibyte character set (MBCS) text | Make sure that text conversions between encodings that use different numbers of bytes per character do not cause buffer overflows, memory leaks, or text truncations. Storage of the same text encoded in UTF-16, UTF-8, and UTF-32, might require buffers of different sizes. | Applications that handle different text encodings
Language-independent data persistence | Applications must store and retrieve documents containing multilingual data, regardless of the language settings of the system where the operation is performed. | All applications that store data in an external persistent storage
Globalization: Locale Awareness
Areas and Items to Test
Details
Applicability
Locale-independent data persistence
Applications must store and retrieve documents containing locale-sensitive data (such as the date, time, and numeric information); check that this data is stored in a locale-neutral form. It must be formatted for display purposes only and must follow the rules of the settings on the reviewer's system.
All applications that store data in an external persistent storage
Adherence to locale standards
Verify that locale-specific data (such as the time, date, currency, and numeric values) can be entered, interpreted, stored, and retrieved according to the rules set by the user locale.

Specifically, verify that:

  • A.M./P.M. symbols are not hard-coded for the time values.
  • Different time and date formats can be used, including different separators.
  • Dates in different calendars can be entered and edited.
  • Positive and negative number formats are recognized correctly.
  • Neither the dollar sign nor any other symbol is hard-coded as a currency symbol.
All applications
Dynamic usage of format separators
Verify that date, time, and numeric values are entered with data separators; make sure the separators are not hard-coded in the application, but rather are defined based on the current user-locale settings.
Applications that allow you to enter formatted data
Paper and envelope sizes
In accordance with the selected locale setting, check that applications pick the appropriate default paper and envelope sizes for correctly formatting information that's going to be printed. However, be sure to let the user override those settings. Verify that applications relay information about the proper default paper size to the printing device.
Applications that format printable information
Measurement-systems independence
Verify that applications that use real physical dimensions for formatting graphic and text data utilize the measurement system (metric or U.S. Customary System) corresponding to the user's locale settings.
Applications that use real physical dimensions for formatting graphic and text data
Localizability
Areas and Items to Test
Details
Applicability
Isolation of the localizable resources
Make sure all UI elements are isolated in the localizable resources. Pseudo-localization should not leave any unlocalized text in the UI. Text that remains untranslated after pseudo-localization most likely is hard-coded in the application's code and creates a localizability problem. In addition to the text that appears in the graphical user interface (GUI), verify that the following text is localizable:
  • Parts of the command-line interface
  • Messages in the Event Log and Error Log
  • Error messages
  • Tool tips
Applications that use text from external resources
Locale-independent handling of resources
Check that static UI text (such as names of dialog boxes, Field labels, tabs, and other things the user cannot change) is displayed in the language of the application's UI. There should be no question marks, default glyphs, or random characters in the UI, regardless of the locale or language settings of the system.
Applications that use separ- ate resource modules for UI text storage
Handling of string dependencies
Applications should not rely on consistency of the resource text in different parts of the system. If the functionality of an application relies on several strings being identical in different parts of the application, either those strings have to originate from a single source, or their relationship should be clearly commented for future localization and support. Make sure that random pseudo-localization does not affect the functionality of the application.
Applications that display UI text from external resources
Language-independence of the file names
Names of external files (such as Help files, satellite DLLs, and database files) that the application uses should not be localized; pseudo-localization should not change them. If pseudo-localization were to change these names, applications wouldn't be able to find their databases.
Applications that open external files to perform some of their operations
Isolation of font properties
Font properties must not be in localizable resources. Applications violating this rule are likely to use an incorrect font face and size if pseudo-localized.
Applications that display text in the GUI
Font-size and display-resolution independence
Verify that applications are not affected if the size of the shell font or screen resolution is changed.
Windowed applications
Localization-independent functionality
Make sure pseudo-localization doesn't affect the following areas:
  • Hot keys
  • Accelerators
  • Menus
  • Responses (like [Y|N]) required by command-line applications
  • Other UI control elements

All of the areas just listed must be accessible on a localized platform, with the input locale matching the UI language of the application.
Applications that have con- trol elements in the UI
UI adjustable to the UI text growth
Growth of the UI text length due to pseudo-localization should not cause text clipping; compensate for this problem by having the UI adjusted manually (by localizers) or automatically (with pseudo-localization). Applications should not assume the UI dimensions are fixed.
Windowed applications that display text in the GUI
Mirroring-awareness in UI composition
Dynamic placement of UI elements might cause the following mirroring problems, seen after pseudo-mirroring is applied:
  • Misplaced controls in dialog boxes
  • Broken check-box display
  • Empty property pages displayed in property sheets
  • Incorrect text alignment
  • Mirroring of the text (text displayed in reverse order)
  • Display not matching the real location of the control (controls are drawn in one place but actually work in another)
Windowed applications
Localizability of nontextual resources
For all nontextual resources (such as images and sounds) that must be localized, make sure clear rules and requirements are defined for the localization.
Applications that incorporate localizable nontextual resources
Run-time string composition
Verify that the dynamic-string composition can hold if the order of parts of the output must be changed. Make sure the composition rules are not based on a single language.
Applications that build strings from pieces at run time
Dynamic retrieval of system-defined names
The applications should not refer to built-in objects of the operating system by names stored in the application: the names of built-in accounts like "Administrator" or folders like "Program Files" must be retrieved dynamically. To check this, pseudo-localize the application and run it on a local- ized platform.
All applications


Chapter 12: Testing Localizability with Pseudo-Localization

When you purchase a new pair of shoes, you don't just walk into the store, tell the clerk what size you want, pay for them, and then walk out without making sure that they fit. By doing so, you most likely would end up spending your hard-earned cash on shoes that are the wrong color, style, and size. Along these same lines, why would you spend hard-earned resources on a particular localization-that might be a poor fit with your application-without first making sure your product is localizable? Pseudo-localization is a valuable, pre-emptive technique for measuring whether the software and content of a product-such as user interface (UI) elements-are localizable. By addressing the results of this technique, developers enable localizers to more easily produce localized versions of a product that have the same high quality of functionality as the original.

While the previous chapter introduced the concept of pseudo-localization within the framework of localizability testing, this chapter offers a more extensive look at its use. You'll see examples of common localizability bugs, as well as some of the features associated with pseudo-localization-including translation of text resources and text-length extension. Also discussed are the pseudo-localization of graphics and audio, of content such as Help files and Web content, and finally pseudo-mirroring for bidirectional (BiDi) languages.

As mentioned in the previous chapter, with pseudo-localization you can apply an automated simulation of the localization process, followed by building and testing these simulated localized versions. Pseudo-localization translates the strings of a product into "pseudo-strings." The resulting "pseudo-language" is designed to test the impact that different aspects of localization have on the product's functionality and appearance. These aspects include translation of strings with the use of different character sets and the growth of string length. Other options for pseudo-localization are the automatic localization of bitmaps (properties or overlaying text) and content; pseudo-mirroring, which simulates localization for right-to-left languages, should not be forgotten either. Since these processes are relatively easy to establish and can be fully automated, they can be applied early in the development cycle.

With products continuously increasing in size and complexity, and with an increasing demand for releasing localized versions of a product soon after releasing the English product, testing pseudo-localized product versions should become a standard part of the overall internationalization process. Fortunately, incorporating this practice is relatively easy. Almost any localization tool available on the market contains add-ons for required pseudo-localization functions or features, offering capabilities to help you find localizability bugs in a more efficient and timely manner. Before you embark on localizability testing with pseudo-localization, it is helpful to be aware of some common localizability bugs that you might encounter.


Categories of Localizability Bugs

The goal of pseudo-localization is to identify weaknesses in the design or source code that can cause functional and cosmetic deficiencies-or localizability bugs-in a localized product. The following is an overview of some common localizability bugs:

There are resources that are not exposed to localization. This problem can occur if the application loads resources from external sources, and the relationship between the application and those resources (and the consequent localization requirements) is not made obvious to the localizer. Take, for example, an application that displays text from a shared library. The library must be localized in order for the application's user interface (UI) to be fully localized.

The format of the file to be localized is either not in a localizable format, or resources are hard-coded in the source code. The only way to modify such resources is to modify the source code or to tamper with the application's internals. As a result, the localization process that deploys standard localization tools will render certain parts of the UI untranslatable.

There are string resources (or parts of string resources) that should not be translated. A common development practice is to separate string resources from the code and store them in resource files. However, string resources used in the UI are not always separated from string resources essential for functionality. Translation of these "functional" string resources, or parts thereof, results in functional bugs. For example, if two applications have to use a named memory-mapped file or semaphore in order to communicate or coordinate execution, both binaries have to use the same name for that shared object. This name should not be translated; translation will add useless workload on the localization team and will not improve the user's experience.

Non-Latin characters cause functional or cosmetic problems. This is essentially the same problem as inability to handle multilingual data, but in this case the problem applies to the resources of the application. Handling of localized resources must work properly with encoding of the text; otherwise, the text will be broken in one way or another.

String-length extension causes buffer overflow. On average, translated strings are longer than English strings. Since text buffers for the text loaded from resources can be allocated statically, translation can cause buffer overflow, which is exposed as functional problems or truncated messages.

Strings are not localized consistently. Functionality is dependent upon string resources being translated consistently. Examples of this are references to folder names, account names, and locale identifiers. Developers are encouraged to store and retrieve the needed value from a central repository, thus removing such dependencies and better enabling cross-language scenarios.

A product cannot be adapted. One element of localization is the adaptation of a product to the local market. This can entail changing fonts, installing additional devices for the local market, installing dictionaries, and so on. Ideally, this adaptation should be a matter of setting locale-specific values in the configuration file and organizing the deployment package.

A product is not mirroring-enabled. Many products are localized into bidirectional (BiDi) languages using either UI flipping or the mirroring technology. Both of the technologies make certain assumptions about the UI. (For more information, see "Pseudo-Mirroring for BiDi Languages" later in this chapter.). If those assumptions do not hold, it is impossible to localize into those languages.

String concatenation and UI creation at run time present problems. The correct localization and translation of resources can be very difficult if the strings and UI are built dynamically at run time. Furthermore, this practice of putting fragmented strings together to create sentences and messages at run time does not always take into consideration the linguistic requirements of the languages into which a product is localized. For example, word order can change from language to language. (See Chapter 7, "Software Localizability Guidelines.") While it is not possible to know if the algorithm that builds text at run time can adapt to grammar rules of other languages, using pseudo-localization can make it possible to display all places where such run-time text manipulations take place.

You can test for and avert functionality and cosmetic problems such as those just listed through pseudo-localization's multiple features. These include translation of string resources, string-length extension, and dialog box stretching.


Most Common Features of Pseudo-Localization

The pseudo-localization process, once again, must be designed to test a product for its ability to be localized, taking into account all possible aspects of localization. The pseudo-language and features of the pseudo-localization process described here are designed to test the robustness of the code with regard to the localizability issues described in the previous section.

One of the most common features in the pseudo-localization process involves translation of string resources. Strings can be translated by prefixing them with extra characters, adding suffixes, or by replacing existing characters. The replacement characters are mostly picked randomly from a predefined set of similar characters from other languages that are determined by the test scenario required. The characters used for replacement often look like the characters they replace in order to ensure readability. For example, Latin "a" can be replaced by one of the symbols from a set: { ä, å, à, à, â, ã, [image]}.

This modification of resource strings will show if the text can be translated, if non-Latin characters are supported by the mechanism that handles resources, and if there is a dependency between the string resources that would require a consistent translation of two different strings. If all string resources are translated, string resources displayed in English will uncover possible hard-coded strings, files that aren't within the scope of localization, or files that are not in a localizable format.

A second common pseudo-localization feature is string-length extension. With this feature, strings can be made longer by appending characters, and string delimiters can be specified so that you can see where a string starts and ends. With string-length extension you can test for buffer overflow and truncated strings; you can also see the text built from several resource strings at run time. For example, suppose you add the left brace symbol ( { ) at the beginning of each string and the right brace symbol ( } ) at the end of it, and that you see the following text displayed by the application:

{string1{string2}} string3 {string4

This particular display means that the whole text was built at run time; "string2" was inserted into "string1;" "string3" was not loaded from a pseudo-localized resource; and "string4" was truncated. A third pseudo-localization feature is that of dialog box stretching. During localization, dialog boxes can be made wider or higher to accommodate the growth in string length or font size and to test for hard-coded dialog box sizes. This is done by modifying the dialog box template.

A fourth feature involves replacement of hot keys and shortcut keys. For instance, you can replace hot keys and shortcut keys (such as Ctrl+B for "Bold") with other characters to test that the keys still function correctly.

Figure 12-1 illustrates a combination of the features just mentioned. In this pseudo-localized version of Notepad, characters are replaced with ones that look similar to them, and strings are extended and surrounded by delimiters. Note that the operating system is not pseudo-localized. As a result, the button text "Yes," "No," and "Cancel" all stay in "correct" English, because this text is defined in the system resources.

Figure 12.1 A pseudo-localized version of Notepad with a message box stretched horizontally.
Figure 12.1 - A pseudo-localized version of Notepad with a message box stretched horizontally.

The message box displayed in this figure is a system component with application-defined text. The application runs on English Windows XP, hence the message and the window caption are pseudo-localized, while button captions are not.


Additional Uses for Pseudo-Localization

Along with the common features of pseudo-localization just discussed, you can pseudo-localize graphics and audio, content such as Help files or Web content, and mirroring properties for BiDi languages. Moreover, commenting models provide a useful way of passing along information-within resource files-to localization teams.

Pseudo-Localization of Graphics and Audio

Some software products make heavy use of graphical and audio resources. When these resources are used to build a localizable UI, pseudo-localization can be applied to them as well. The technique for this pseudo-localization depends on the project. For example, graphic elements can be modified to display their file name on top of the graphic itself. Also, audio files can be extended to contain their file name character by character.

Pseudo-Localization of Content (Help Files, Web Content)

Bugs found during content localization are similar to those seen in the localization of other components: incorrect display of characters, hard-coding of strings, incorrect BiDi text flow, or mirroring-enabling. Since the volume of content can be significant, it is important to be able to test the content files for these problems early in the product cycle. As mentioned earlier, pseudo-features are often provided as add-ons to existing tools for content localization.

Pseudo-Mirroring for BiDi Languages

Creating truly mirrored applications is described in Chapter 8, "Mirroring." To test if a product is correctly enabled for BiDi or mirroring, you can simulate the localization process by changing all file and dialog box properties in the same way the localizer would. By changing BiDi or mirroring properties only and not pseudo-localizing other resources, you can create builds to test BiDi and mirroring-enabling specifically. Figure 12-2 shows a pseudo-mirrored version of Microsoft Paint. Note the images and UI elements that are mirrored. Also worth noting is the English text that is displayed in a right-to-left order, resulting in punctuation marks that appear to the left of a sentence.

Figure 12.2 A pseudo-mirrored version of Microsoft Paint.
Figure 12.2 - A pseudo-mirrored version of Microsoft Paint.

Commenting Models

Many resource formats support a commenting model of some kind. Information on whether strings can be localized, whether limitations on the string length exist, or simply regular comments can be added to the resource files and passed on to localization.

Alternatively, when no commenting model is available, agreements can be made between development and localization teams on how to document information. Examples of this are the use of certain identifier ranges or names for non-localizable items, the separation of localizable elements in separate resource dynamic-link libraries (DLLs), and the maintenance of separate instruction files attached to resource files.

Based on the information that the development team provides, use the pseudo-localization tools to create pseudo-localized strings. By doing so, you can prevent many potential localizability bugs from entering a pseudo-localized build. The information can also be passed on to the different localization teams, and localized strings can be verified for compliance with the rules that developers establish.


Summary

With pseudo-localization you first apply an automated simulation of the localization process, followed by building and testing these simulated localized versions to find localizability bugs. Some common bugs include resources that are not exposed to localization; file for mats that are either not in a localizable format, or that have resources that are hard-coded in the source code; string resources that shouldn't be translated; non-Latin characters that cause functional or cosmetic problems; products that cannot be adapted; products that are not mirroring-enabled; string-length extension that causes buffer overflow; string concatenation and a run-time UI that present problems; and strings that are not localized consistently.

Among the most common features in the pseudo-localization process are translation of string resources, string-length extension, dialog box stretching, and replacement of hot keys and shortcut keys with other characters. In addition, you can use pseudo-localization for graphics and audio files, content such as Help files and Web content, and for pseudo-mirroring when working with BiDi languages. By using pseudo-localization in all its capacities, your localizability test will catch potential problems and help ensure high-quality localized products.


Part VI: Tools and Technologies

When a house is being constructed, it would be inconceivable for those involved in the project not to gather and utilize a wide assembly of tools that enable, accelerate, and ease the entire process. Without such things as drills, nail guns, hammers, chisels, saws, jigs, and sometimes scaffolds, certain construction tasks will be more difficult if not impossible to complete, and they will be less cost-effective. Likewise, designers and developers of world-ready software should be thoroughly equipped to handle all the various aspects related to making software usable in the international arena.

Because Microsoft realizes the significance of internationalized products, it has made available all the parts needed to create world-ready solutions at all levels of deployment. In addition to the invaluable resources, techniques, and technologies discussed throughout this book—including Unicode, numerous application programming interfaces (APIs), and Microsoft Windows globalization services—there is yet another assortment of resources at your disposal, which is the focus of Part VI, "Tools and Technologies." Part VI contains a sampling of tools and technologies that can expedite and simplify the process of developing world-ready software. Although the discussion of each resource could easily be expanded into a book in itself, the primary purpose of these chapters is to present a high-level glimpse of the role these tools and technologies play, and to point you to further information regarding their usage.

In general, each chapter is divided into similar components: "Overview of the Technology," (or "Overview of the Tool"), "Availability," "International Features" (when applicable), "International Best Practices" (when applicable), and "Additional Resources." Some of these tools and technologies, such as Microsoft Layer for Unicode (MSLU), are inherently world-ready and, consequently, each feature is regarded as international in nature. For this reason, these features are not specifically mentioned as separate international components. Similarly, the mere act of using other tools and technologies such as the Microsoft Windows Text Services Framework (TSF), Uniscribe, or Microsoft Visual Studio .NET represents an international best practice in itself. When there are issues or alternatives involved with using a particular resource, as in the case of OpenType fonts, these topics are also presented. As a final note, although the .NET Framework platform is discussed throughout the book, it is included in Part VI to underscore its vital role in the development of world-ready applications and to provide a more general look at the platform as a whole.


Chapter 13: Graphics Device Interface Plus (GDI+)

This chapter discusses the benefits of using GDI+ when creating world-ready applications. Included is an overview of what this technology provides, where it is available, and what international features and international best practices are associated with it. Finally, the chapter will supply you with additional resources regarding this technology.


Overview of the Technology

GDI+ is a graphics device interface library that ships with Microsoft Windows XP and Microsoft Windows .NET Server. GDI+ includes impressive support for two-dimensional (2D) graphics and imaging, and optimal support for text and fonts.

GDI+ is the successor to GDI, the graphics device interface included with earlier versions of Windows. While Windows XP and Windows .NET Server continue to support GDI for compatibility with existing applications, if you are writing new applications, you should take advantage of the additional features and improved programming mode that GDI+ offers, as examined in the next section.

As was true for GDI, one of the many benefits of using GDI+ is that it insulates the application from the graphics hardware, thus allowing you to draw objects to the screen or printer without having to concern yourself about the details behind different device technologies. GDI+ application programming interfaces (APIs) are exposed through a set of C++ classes and as part of the Microsoft .NET Framework.

What's New in GDI+?

Glossary
  • Linear gradient brush: Can be horizontal or vertical. For example, a horizontal gradient brush can be configured to change color as you move from the left side of a figure to the right side.
  • Path gradient brush: A path gradient brush can be configured to change color as you move from the center of a figure toward the boundary.
  • Cardinal splines: A sequence of individual curves joined to form a larger curve.

As stated, GDI+ has many capabilities that GDI does not offer. In 2D graphics, for instance, GDI+ expands GDI support by providing linear gradient and path gradient brushes for filling shapes, paths, and regions. Brushes can also be used to draw lines, curves, and paths. In addition, GDI+ provides support for scalable regions, alpha blending, cardinal splines, and transformations. Furthermore, GDI+ supports multiple image formats-including .bmp, .png, .gif, and .png-which allows you to load, save, and manipulate different types of images.

There have been significant improvements to text and font support since GDI. For example, GDI+ supports resolution-independent layout, guaranteeing that text layout and line breaks will not change as the output-device resolution changes, a common pitfall when using GDI. You are probably familiar with this problem if you ever used GDI to draw text to devices that have different resolutions, such as to screens and printers. As the application switches between output devices with different resolutions, there is a greater chance of clipping because paragaraphs get longer as the line-breaking positions change.

Even if you are writing a user interface (UI) widget (such as a drop-down menu) that works only on the screen and you never need to deal with a printer, you will still benefit enormously from GDI+ resolution-independence. Higher-resolution screens of 150 to 200 dots per inch (DPI) are just around the corner. With the resolution-independent layout that GDI+ provides, you'll be ready for this change.

GDI+ incorporates a cleaner and much improved font management architecture that allows side-by-side installation of the same font by using private font collections. In addition, GDI+ supports many new scripts like Gujarati, Gurmukhi, Kannada, and Syriac.

Programming Model

Core to the GDI+ programming model is the concept of a graphics surface, represented by the Graphics class. The Graphics classplays a similar role to the GDI device context. This class represents both the capabilities of a particular output device and the attributes that specify how items will be drawn on that output device, whether it is a screen or a printer. The main difference in the programming model between a GDI device context and GDI+ Graphics objects is that Graphics objects are not tied to a font, pen, brush, path, or image. As a result, Graphics has no SelectObject method to select a font in the device.


Availability

GDI+ is available starting with Windows XP and Windows .NET Server. It is also available for download from the Microsoft Developer Network (MSDN) at http://msdn.microsoft.com.


International Features

Glossary
  • Typography: The process of displaying text in a variety of fonts, sizes, and styles.

One of the most significant features of GDI+ is its support for handling typography-that is, for dealing with the numerous kinds of fonts and scripts encountered in multilingual computing. GDI+ was designed to be language-agnostic and script-agnostic. Unlike GDI, it has no implicit assumptions about the relation between a character's code point and the glyphs used to represent a character in the font. In GDI+ the relation between a character's code point and the glyphs used in the font is many-to-many, which is a very important feature for multilingual computing. For example, two Unicode code points can be represented by one glyph or vice versa. GDI+ also offers optimal font support through various Graphics classes and methods, as discussed in the section that follows.

Another international feature of GDI+ that will make your job much easier is font fallback. Font fallback frees the application and the user from having to know about the scripts supported by each font on the system. If the application tries to display characters from a script that is not supported by the selected font in DrawString, GDI+ automatically switches to a font that can display those characters. Font fallback is also a great benefit for you as a developer, since it spares you from having to know about the scripts that each user might be working with. (For more information, see "Font Fallback" later in this chapter.) Other international features of GDI+ that will make your job less complicated are anti-aliasing and ClearType.

Typography Handling in GDI+

As mentioned, GDI+ is adept at handling typography because of its many-to-many relation between character code points and the glyphs used in a font. When dealing with typography in the context of GDI+, there are several key ideas to bear in mind. You must use Unicode for your text input. Also, you should take advantage of the multiple classes and methods available for font and text support, in addition to leveraging GDI+ support for a host of different scripts.

Unicode

GDI+ doesn't support text that is based on code pages; GDI+ APIs are entirely Unicode-based. Even when running on Microsoft Windows 98 and Microsoft Windows Millennium Edition (Me), GDI+ expects all the input to be in Unicode. If your application is still using code pages, you need to convert the text to Unicode before calling Graphics.DrawString.

The Use of Classes and Methods

GDI+ provides several Graphics classes and methods that allow your application to:

  • Enumerate fonts installed on the system.
  • Create font collections that are private to the application.
  • Obtain font design metrics.
  • Draw text horizontally or vertically.
  • Apply special formatting such as horizontal and vertical alignment, tab stops, and hot-key prefixes to the text.

Scripts and Writing Systems

To accommodate the vast number of writing systems in use today, GDI+ supports the same scripts as Windows XP, including the following:

  • Latin, Greek, and Cyrillic scripts. These scripts are used for English, French, Spanish, German, Greek, Russian, and many other languages' writing systems.
  • East Asian scripts. These scripts are used for Chinese, Japanese, Korean, Vietnamese, Thai, and Ancient Hangul.
  • Bidirectional scripts. These scripts are used in the Arabic writing system, which includes languages like Farsi and Urdu. They are also used in the Hebrew writing system, which includes languages like Hebrew, Yiddish, and Syriac.
  • South Asian scripts. These scripts are used in Devanagari, Tamil, Gujarati, Gurmukhi, and Kannada.

The fonts installed on the specific computer on which your application is running determine which scripts your GDI+ application can display. For example, if your application is running on Windows XP or later, you can display text from any of the scripts just listed1. Figure 13-1 demonstrates output from one of the Microsoft Windows Forms samples within the .NET Framework. The figure shows text from Arabic, Japanese, and Latin scripts, in addition to the enhanced graphics brushes that the application can apply to text.

Figure 13.1 Output from one of the Windows Forms samples within the .NET Framework.
Figure 13.1 - Output from one of the Windows Forms samples within the .NET Framework.

Font Fallback

One of the issues that arises when dealing with Unicode text is which font the application should use to cover all Unicode ranges. A given script in Unicode has specific typographic features and requirements that in many cases are in sharp contrast with those of other scripts. Since Windows XP and Windows .NET Server don't ship any font that covers all of Unicode, GDI+ uses font fallback. If the application attempts to display characters that are not supported by a font, GDI+ will automatically switch-or fall back-to another font on the computer that supports the script.

Suppose that the application is trying to display Hindi text using Arial font, which contains no glyphs for the Devanagari script. GDI+ will internally switch to the Mangal font, which does support Devanagari. Table 13-1 details which font GDI+ uses to fall back for a given script.

Table 13-1 Fallback fonts that GDI+ uses for given scripts.

Script | Font Face Name
Armenian | Sylfaen
East Asian Scripts (including Simplified Chinese, Traditional Chinese, Japanese, and Korean) | Microsoft Sans Serif2
Georgian | Sylfaen
Gujarati | Shruti
Gurmukhi (Punjabi) | Raavi
Hindi | Mangal
Kannada | Tunga
Syriac | Estrangello Edessa
Tamil | Latha
Telugu | Gautami
All other scripts (including Latin, Cyrillic, Greek, Arabic, Hebrew, Thai, and Vietnamese. | Microsoft Sans Serif (on Windows 2000 and Windows XP) Arial (on Microsoft Windows 95, Windows 98, Windows Me, and Microsoft Windows NT 4)

Clearly, if the original font the user or the application specified doesn't contain glyphs for particular characters, and if no fallback font is available on the system, GDI+ will not be able to perform font fallback. Instead, GDI+ will display the missing glyph as an empty square box. (For more information on font fallback, see Chapter 5, "Text Input, Output, and Display.")

Anti-Aliasing and ClearType

Glossary
  • Anti-aliasing: A software technique for smoothing the jagged appearance of curved or diagonal lines caused by poor resolution on a display screen.

Another benefit of using GDI+ is that it provides various quality levels for displaying text. Higher-quality text improves the readability of the text on the screen, which is important for applications designed for intensive reading. Typically, higher-quality rendering takes more processing time than lower-quality rendering.

In addition to traditional anti-aliasing supported by GDI, which is grid-fitted to pixel boundaries, GDI+ supports subpixel gray anti-aliasing and Microsoft's new text display technology called "ClearType." (ClearType technology is only supported on Windows XP and later.) ClearType font smoothing improves text readability on color LCD monitors that have a digital interface, such as on laptop screens and high-quality flat desktop displays. Readability on CRT screens is also somewhat improved.

The following code sample draws text with two different quality-level settings:

FontFamily fontFamily(L"Times New Roman");
Font        font(&fontFamily, 32, FontStyleRegular, UnitPixel);
SolidBrush  solidBrush(Color(255, 0, 0, 255));
WCHAR       string1[] = L"SingleBitPerPixel";
WCHAR       string2[] = L"AntiAlias";

graphics.SetTextRenderingHint(TextRenderingHintSingleBitPerPixel);
graphics.DrawString(string1, -1, &font, PointF(10.0f, 10.0f),
   &solidBrush);

graphics.SetTextRenderingHint(TextRenderingHintAntiAlias);
graphics.DrawString(string2, -1, &font, PointF(10.0f, 60.0f),
   &solidBrush);

Figure 13-2 shows the output of the preceding code.

Figure 13.2 Output of code that draws text with two different quality-level settings.
Figure 13.2 - Output of code that draws text with two different quality-level settings.

You have seen the many features that GDI+ offers to foster the development of world-ready applications. By following the best practices discussed in the next section, you can leverage all the capabilities of GDI+ even further.


International Best Practices

Whether you are using GDI+ directly or through Windows Forms based on the .NET Framework, there are a few best practices that you should follow.

Consider the Differences Among Various Scripts

Do not design your application to make assumptions on how a font will present the code points of a particular string. For example, in simple scripts like English and other Latin-based scripts, characters' positions progress from left to right as they are typed, and each individual character is typically represented by a single glyph. It is a common pitfall for developers to assume that other scripts will function in the same manner, so they sometimes design their application so that it displays characters entered to the right of the caret position.

This type of application will not function properly for a user dealing with complex scripts. In Indic languages, for instance, characters often combine to form ligatures, and characters can change position with respect to each other at display time.

Do Not Display One Character at a Time

Your application is in the best position to support any script (even the ones you don't know about) when you display the entire line of text, instead of trying to optimize by displaying one character at a time. Displaying the entire line allows characters to change their shape and position based on the context of the line, guaranteeing proper script handling and language handling.


Additional Resources

For more information on GDI+, see the following resources:

  1. http://msdn.microsoft.com. Provides general reference information on GDI+.
  2. Petzold, Charles. Programming Microsoft Windows with C#. Redmond, WA: Microsoft Press, 2001. ISDN 0-7356-1370-2. Includes topics such as text output, fonts, splines, images and bitmaps, paths, regions, and clipping.


Summary

Available as part of Windows XP and Windows .NET Server, GDI+ is a valuable technology because of its excellent support for two-dimensional (2D) graphics and imaging and its optimal support for text and fonts. While GDI+ does not replace GDI as a graphics device interface, GDI+ does offer considerable advantages over GDI. GDI+ expands GDI support for 2D graphics by providing linear and path gradient brushes for filling shapes, paths, and regions. GDI+ text and font support has also been greatly improved over GDI. Unlike GDI, GDI+ supports resolution-independent layout, guaranteeing that text layout and line breaks will not change as the resolution of the output device changes.

International features of GDI+ include its ability to handle various scripts and fonts. For example, two Unicode code points can be represented by one glyph or vice versa. Also, Graphics classes and methods allow your application to enumerate fonts installed on the system, create font collections that are private to the application, and obtain font design metrics. Additionally, these classes and methods let you draw text horizontally or vertically, and allow you to apply special formatting like horizontal and vertical alignment, tab stops, and hotkey prefixes to multilingual text. Another international feature of GDI+ is its use of font fallback. Finally, anti-aliasing and ClearType enhance readability of text on the screen.

When working with GDI+, you should consider the differences that exist among various scripts. While character reordering or characters that combine are not issues in Latin-based text, they are an important component of other scripts, such as complex scripts. You will need to design your application with these differences in mind. Also, rather than displaying one character at a time, it is better to display an entire line of text to account for character reordering and for characters that combine to form ligatures.


Chapter 14: Hypertext Markup Language (HTML)

This chapter discusses the benefits of using HTML when creating world-ready applications. Included is an overview of what this technology provides and what international features are associated with it. Finally, the chapter will point you to information elsewhere in this book regarding international best practices, as well as supply you with additional resources regarding this technology.


Overview of the Technology

Glossary
  • Element: The basic unit of information in an HTML or XML document. Elements are arranged hierarchically to define the overall document structure.
  • Attribute: A parameter that defines a special property of an HTML element. Attributes are specified within start tags. For example, <IMG SRC = "image.gif"> means that the element IMG has an attribute called "SRC," which is assigned the indicated value.

HTML, the standard on which the World Wide Web (WWW) is founded and a descendant of Standard Generalized Markup Language (SGML), provides the basic mechanism by which documents on the Web are presented and linked to each other. Tim Berners-Lee is credited with the first proposals for creating HTML in 1989. His goal was to simplify the maintenance and the publishing of research documents on the computers of the Centre Européen pour la Recherche Nucléaire (CERN) in Geneva, Switzerland where he worked, while also making these documents more accessible to the rest of the research community. Berners-Lee and colleague Robert Caillau wrote the first program to render HTML documents in 1991.

Berners-Lee derived the basic rules for HTML by using the International Organization for Standardization (ISO) 8879 standard of 1986 for SGML. For instance, the tags used in HTML to denote the elements of a document come directly from SGML. A tag is a keyword enclosed within a pair of angle brackets (<, >), whereas elements are enclosed within a start tag and an end tag. The end tag carries the same keyword as the start tag, but is preceded by a slash (/). Here is an example:

<title>This is the title of the document</title>

SGML was designed with human editors in mind who would manually compose the text and include tags to identify the semantic functions of each text element (such as <title> or <body>). That is why HTML documents are readable with any plaintext editor like Notepad. However, the layout of most of today's HTML documents has become so complex that the amount of markup within an HTML document easily surpasses the amount of actual text.

Originally, HTML was meant to represent text within a document structure, rather than accounting for any formatting or attributes of the presentation. Nevertheless, as the implementers of Web browsers and the authors of HTML documents were looking for methods to control the visual representation of HTML documents, more and more tags were added allowing exactly that. As a result, HTML has become a very rich markup language that allows the representation of the most complex layouts.

Because most of the documents that CERN maintained and published were written in multiple Western European languages, a basic international feature was present in the original versions of HTML-namely, the capability to render documents written in any Western European language that uses the Latin 1 character set. Latin 1 is, in fact, the default character set for HTML documents.

Today's popular Web browsers implement HTML in the form that is standardized by the World Wide Web Consortium (W3C). The first version of HTML standardized by the W3C was HTML 4, which included a host of international features. These features far surpassed those present in earlier versions of HTML and, for the first time, bidirectional support was added, as required for documents written in Hebrew, Arabic, or Farsi, for example. (For more information on international features, see "International Features" later in this chapter.)

Over the years, HTML has been enriched with the ability to add dynamic logic to documents via scripting. The most popular languages to apply script-ing within HTML are Microsoft JScript (also known as "ECMAScript Edition 3"), Microsoft Visual Basic Scripting Edition (VBScript), and JavaScript.

Microsoft Internet Explorer makes use of the dynamic-link libraries (DLLs) Shdocvw.dll and Mshtml.dll (associated with MSHTML, Microsoft's own superset of HTML tags) to handle all its reading and rendering of HTML documents. Any Microsoft Windows application can host either one of these libraries and use its interfaces for managing HTML or fragments of an HTML document. Both DLLs allow access down to the individual elements of the document. Applications can use the simple browser control of Shdocvw.dll to implement what is essentially another version of a Web browser. Alternatively, applications can directly host Mshtml.dll to gain access to and manipulate the document's elements. (For more information on hosting MSHTML and its associated DLL, go to http://msdn.microsoft.com/workshop/browser/hosting/Hosting.asp.)

Before taking a closer look at HTML's international features, there are some basic terms with which you should be familiar. These terms-used frequently (and sometimes interchangeably) throughout various Request for Comments (RFCs), ISO standards, and other documents-include "encoding," "character encoding," "character set" and "charset." (For more information on these terms, see Chapter 3, "Unicode." )


International Features

This section probes deeper into HTML's international capabilities, all of which will assist you in the creation of world-ready software. As stated, a basic inter-national element was present in the original versions of HTML, though with HTML 4 numerous international features were added. Some of the features that assist in dealing with different character sets and encodings include the HTML charset declaration (see Chapter 3), character entity references, numeric entity references (see Chapter 3), form submission, URL encoding, cascading style sheets (CSS), and the LANG attribute.

Character Entity References

Character entity references provide symbolic names (or entities) for a set of characters and are defined by the HTML standard. Character entity references always start with the character "&," followed by the symbolic name and a semi-colon. The four entity references that you should be familiar with are:

  • &lt; for the character "<" (less than)
  • &gt; for the character ">" (greater than)
  • &amp; for the character "&" (ampersand)
  • &nbsp; for a nonbreaking space character

The first three entity references-&lt;, &gt;, and &amp;-are essential because otherwise the characters they represent cannot be rendered in an HTML document. HTML 4 defines 248 additional entity references, covering the Greek and Latin 1 alphabets as well as additional punctuation characters. A standards-compliant browser such as Internet Explorer 5 or later will render any of these references. However, for better compatibility, it is recommended that you use the character itself within the document, rather than the entity. There is no reason to use any entity references other than the four listed previously. The additional 248 entity references exist merely to allow human editing of the HTML source, such as when someone has a keyboard that does not contain certain characters.

Other examples of entity references include the following:

  • &euro; for the euro sign " [image] "
  • &umlaut; for the character "ä"

Form Submission

HTML offers a method to submit text input from the browser to a Web server via the FORM element. The FORM element can send the text input by using the HTTP methods GET or POST. With either method, the encoding of the submitted text data follows the encoding of the document in which the FORM element is embedded. For example, if the document encoding is ISO 8859-1, the character "Ä" would be submitted as 0xC4.

The HTML standard leaves undefined what would happen if a user entered a character outside ISO 8859-1 in the example just given. Internet Explorer will submit characters that have been entered into a FORM element-and that are outside the document's encoding-as decimal numeric character references. For example, if the document encoding is ISO 8859-1, the Greek character [image] (sigma) would be submitted as "&931;".

A good method to avoid ambiguity in form submissions is to always encode HTML documents in UTF-8 or UTF-16. This way, the receiving application or service does not have to deal with translating numeric character references back to actual characters, but rather can process all possible returns the very same way.

The FORM element in HTML 4 allows you to specify an attribute called "Accept-Charset." The function of this attribute is to restrict the list of encodings acceptable to the server. Internet Explorer 5 and later will respond only to a value of UTF-8. If the browser finds

<form accept-charset="utf-8">

Internet Explorer will submit the form data in UTF-8, regardless of the document's character encoding. With this method, it is possible to post the pages in any legacy encoding for maximum compatibility, but the server-side script can process UTF-8 for data submitted by browsers that respond to Accept-Charset. Whether a document is encoded in UTF-16 or UTF-32, the data will be submitted in UTF-8.

Table 14-1 summarizes how data of various document encodings will be submitted.

Table 14-1 Encoding used for submitting data.

Document Encoding | Accept-Charset Value | Encoding of Submitted Data
UTF-8 | No relevance | UTF-8
UTF-16 | No relevance | UTF-8
UTF-32 | No relevance | UTF-8
Any other charset | UTF-8 | UTF-8
Any other charset | Not specified | Same as document encoding

URL Encoding

RFC 1738 is the reference containing standards for URLs. It declares that only 7-bit characters can appear in URLs. This declaration allows the complete US-ASCII character set to appear inside a URL, with the exception of the characters reserved for the HTTP protocol, such as the colon (:), the ampersand (&), the pound sign (#), the question mark (?), the hyphen (-), and the space character. RFC 1738 allows characters outside the US-ASCII character set to be expressed using a percent character (%) followed by two hexadecimal digits denoting an 8-bit character, in a mechanism known as "%hh escaping." For example, "%D6" represents the Latin capital letter "O" with diaeresis in the ISO 8859-1 character encoding. RFC 1738 does not specify which character encoding should be applied to these 8-bit characters. In the final analysis, the browser and server must basically agree on the character to use for the elements of the URL.

Figure 14-1 illustrates the elements of a URL, shown using the HTTPS protocol, and the remainder of this section explains these elements one by one.

Figure 14.1 The elements of a URL.
Figure 14.1 - The elements of a URL.

The element called "pr" is the protocol identifier. The protocol defines the semantics of the remaining elements of the URL. In the previous figure, the secure version of HTTP is specified. All names of protocols are composed of a subset of the US-ASCII character set. No other characters are allowed.

The element called "host" gives the domain name system (DNS) name of the computer to which this request is sent. The DNS is standardized to use a subset of US-ASCII for domain names, although effort is being made to extend the DNS so that it handles characters outside the US-ASCII character set. These efforts have not yet yielded a worldwide-accepted implementation standard. Most current proposals are modeled after Row-based ASCII-Compatible Encoding (RACE), which encodes a string of Unicode code points as a relatively short string of 7-bit characters. Typically, a unique signature (for example, "bq--") precedes a RACE-encoded string.

The element called "path" identifies an object on the server given by the host name. In this example, the addressed object is an ASP page ("disv1.asp") within a certain folder in the host's file system. If escaped using the %hh notation, 8-bit characters are allowed and not uncommon for the path. There is no general standard stating the character encoding to use for the path. In practice, UTF-8 has been deployed; it generally has the best chance of succeeding in finding the desired object. Both Internet Explorer and Microsoft Internet Information Services (IIS) have standardized on using UTF-8 for the path.

The element called "query" passes one or more parameters to the object given in the path. Again, %hh-escaped 8-bit characters are allowed. The character encoding of the query is independent of any other elements of the URL, and it is up to the browser and author of the server script to agree upon this encoding. It is recommended that the browser tell the server script what the character encoding of the query is, such as by appending an extra field into the query-for example, "_charset_=utf-8."

Another useful feature in conjunction with HTML involves cascading style sheets. This feature lets you specify the desired formatting for an HTML document.

Cascading Style Sheets

CSS is designed to separate instructions for the visual representation of data from the data itself. A style sheet can be embedded inside an HTML document, or can be referenced as an external document that the browser fetches from the server. Of international relevance are three elements of style sheets: the encoding of the style sheet document itself, and the referencing of fonts by both their names and their sizes. Style sheets enable Web site authors to determine and modify the visual appear ance of a large set of documents on a Web server-or even across a range of Web servers-and across a multitude of languages to achieve a common corporate identity.

If the style sheet is shared between languages using the Latin script and languages using ideographic scripts like Chinese or Japanese, the author of the style sheet needs to allow the localizer or translator to adjust the font names to the language. This adjustment is necessary because typically the fonts for Latin-based languages do not contain ideographs. The reverse is not true; the Latin glyphs in ideographic fonts often do not meet the visual appearance standards required by Western readers. In addition, the ideographic fonts that shipped with the Chinese version of Microsoft Windows 95 and Microsoft Windows 98 use a native (Chinese) name, whereas all other language versions of these two operating systems refer to the same font by a name in Latin characters. This is also true for the fonts that shipped in the Japanese and Korean versions of Windows 95 and Windows 98. It is only since Microsoft Windows 2000 that fonts can be specified either natively or by using Latin characters, regardless of the language version of Windows 2000.

The author of the shared style sheet should also allow the localizer or translator to adjust the font size. Languages using ideographic scripts will need to increase the font size by 1 or 1.5 points to achieve the same look and feel as the Latin-based language versions of documents that use the same style sheet. Increasing the font size will also maintain readability.

A style sheet can either be embedded into the HTML document or be referenced as an external document. If embedded into the HTML document, the character encoding will be the same as that of the parent document. For an external style sheet, CSS defines the @charset rule. The @charset rule takes a string as an argument, specifying the character encoding to use. This rule needs to be placed at the very beginning of the document and can appear only once.

Here is an example:

@charset "ISO-8859-1";

This string tells the browser to interpret characters within the cascading style sheet as ISO 8859-1. (For more information on CSS, see "Setting Direction with CSS" later in this chapter, as well as the HTML AutoLayout [HAL] document in the Misc subdirectory on the companion CD.)

The LANG Attribute

Another internationalization feature of HTML is the LANG attribute, which was defined by HTML 4 in order to indicate the language of an element. LANG refers to a "natural" language that humans use in order to speak or write; in other words, the LANG attribute excludes computer languages. (In contrast, the LANGUAGE attribute to the script tag refers exclusively to computer languages.) The allowed values for the LANG attribute follow RFC 3066 (which updated RFC 1766). A LANG attribute typically consists of a two-letter ISO 639 language identifier (or primary code), such as "en" for English, "de" for German, "ja" for Japanese, or "hi" for Hindi. The language identifier can be (but does not have to be) followed by a two-letter country code (or subcode) in accordance with the ISO 3166 set of country/region identifiers. Formally defined, the "LANG value" is a language code consisting of a primary code, followed by a dash (-) and possibly an empty set of subcodes.

For languages that do not have a language identifier or country code listed in ISO 639 or ISO 3166, respectively, the Internet Assigned Numbers Authority (IANA) can define additional languages (as it has recently done for sign languages). These particular languages have a language identifier that consists of more than two letters. Language codes starting with "x-" are user-defined and are not standardized.

The LANG attribute also helps with lexical analysis of the text, such as when checking spelling or when using speech synthesis. LANG is not used to specify the directional layout properties of the HTML document. Some browsers, including Internet Explorer, take the LANG attribute into account when choosing an appropriate font to render a section of text. In addition to using the LANG attribute for indicating the language of an element, there are certain language properties that are handy in determining what language a particular user prefers.

Other Language Properties

The Accept-Language field and the properties exposed in the HTML Object Model of Internet Explorer 5 and later are ways to determine user language preferences. Accept-Language accomplishes this by assigning quality values to a particular language. Besides determining user language preferences, the properties exposed in the HTML Object Model are useful for determining operating-system settings such as the current user locale or current system locale. (For more information on determining user language preferences and system settings, see Chapter 4, "Locale and Cultural Awareness." )

Bidirectional Layout

Like many other international features in HTML, version 4 of that standard introduced rules for the definition and handling of bidirectional text. While text that uses the Arabic, Hebrew, or Farsi script is written from right to left, documents employing these scripts can embed elements from other scripts, such as company names in Latin script that run from left to right. Because both directions are allowed within the same body of text, the arrangement is called "bidirectional." The LANG attribute, discussed in "The LANG Attribute" earlier in this chapter, has no influence on the directionality of text. The HTML DIR (direction) attribute is used to indicate the base directionality of an element: "RTL" to indicate right-to-left direction and "LTR" to indicate left-to-right direction. (For more information on bidirectional text, see Chapter 5, "Text Input, Output, and Display.") In addition to the DIR attribute, other elements that can affect the directionality of text are the bidirectional override (BDO) element and direction setting with CSS.

The BDO Element

The BDO element is used to override the shaping of text. However, the DIR attribute must be used with this element to invoke the proper direction override. The BDO element is useful for overriding the Unicode bidirectional algorithm, such as if you need to mix alphabetic and numeric characters (as when representing part numbers of a computer, for example).

Using <BDO dir="rtl"> will render the text "Mirror" as "rorriM." Using <BDO dir="ltr"> will have no effect on Latin text because the Unicode algorithm has already assigned left-to-right directionality to Latin text.

Setting Direction with CSS

Glossary
  • Strong character: A character from which text direction can be determined.
  • Left-to-right embedding (LRE) mark: Signals that a piece of text is to be treated as embedded left to right. For example, an English quotation in the middle of an Arabic sentence could be marked as being embedded left to right. (LRE affects word order, not character order.)
  • Right-to-left embedding (RLE) mark: Signals that a piece of text is to be treated as embedded right to left. For example, a Hebrew phrase in the middle of an English quotation could be marked as being embedded right to left. (RLE affects word order, not character order.)
  • Pop directional formatting (PDF): Terminates the effects of the last explicit code (either embedding or override) and restores the bidirectional state to what it was before the last LRE, RLE, right-to-left override (RLO), or left-to-right override (LRO) control characters.

CSS has a feature similar to the DIR attribute. In CSS, direction is set through the use of two style properties: Direction and Unicode-bidi. The following sections, which explain both these properties, stem directly from the CSS2 specification.

The Direction Property

This property specifies the direction of base writing (in terms of blocks of text), as well as the direction of embeddings and overrides for the Unicode bidirectional algorithm. In addition, the Direction property specifies the direction of column layout in tables, the direction of horizontal overflow, and the position of the last line of text in a block (such as the last line in a paragraph).

Values for this property have the following meanings:

  • ltr : This signifies left-to-right direction.
  • rtl : This signifies right-to-left direction.
  • inherit : This property takes the same value as the property for the element's parent.

For the Direction property to have any effect on inline-level elements, the Unicode-bidi property's value must be embed or override.

Note
The Direction property, when specified for table column elements, is not inherited by cells in the column since columns don't exist in the document tree. Thus CSS cannot easily capture the DIR attribute inheritance rules described in HTML 4, section 11.3.2.1.

The Unicode-bidi Property

Browsers complying with the Unicode bidirectional algorithm-for example, Internet Explorer-will display characters in the correct writing direction automatically. In cases where authors must influence the rendering with explicit directional markup, the Unicode-bidi property can be used to signal that an element opens an explicit embedding or directional override.

The values for the Unicode-bidi property are:

  • normal : The element follows implicit BiDi rules. This means that implicit reordering works across element boundaries based on the Direction property and on strong characters in the text.
  • embed : The element explicitly opens an additional embedding and is equivalent to using an LRE or RLE mark in the code. The agent must terminate the explicit embedding at the end of the element (using PDF).
  • bidi-override : The element explicitly forces the characters to be treated as strong characters in the direction specified by the Direction property. This is equivalent to using the BDO element. The agent must terminate the explicit override at the end of the element (using PDF). This is used for special cases, such as part numbers.
  • inherit : The property takes the same value as the property for the element's parent.

The Direction property has an implicit effect on inline elements when applying the Unicode bidirectional algorithm. The Unicode-bidi property is only required when explicit markup is required to achieve the desired result.

The following HTML example demonstrates that the Unicode-bidi property is required to achieve proper text layout with inline elements based on the Unicode bidirectional algorithm. It also illustrates an important design principle: Web page designers should take bidirectionality into account, both in the language proper (elements and attributes) and in any accompanying style sheets. The style sheets should be designed so that bidirectional rules are separate from other style rules.

<HTML DIR=RTL>

<HEAD>
<TITLE>Direction with Styles</TITLE>
<META charset=windows-1256>
<STYLE>
   DIV.arabic   {direction: rtl; unicode-bidi: normal;}
   DIV.english  {direction: ltr; unicode-bidi: normal;}
   SPAN.arabic1 {direction: rtl; unicode-bidi: normal;}
   SPAN.arabic2 {direction: rtl; unicode-bidi: embed;}
</STYLE>
</HEAD>

<BODY>
<DIV class=arabic>
   <P>2   english3 5
   <BR>6   <B>7   </B>8  </P>
</DIV>
<hr>

<DIV class=english>
   <P>english9 english10 english11 13   12
   <BR>english14 english15 english16</P>
   <hr>

   <P>The following line has: Span.arabic1
      {direction: rtl; unicode-bidi: normal;}
   <P>english17
      <SPAN class=arabic1>18   english19 20  </SPAN>
   </P>
   <hr>

   <P>The following line has: Span.arabic2
      {direction: rtl; unicode-bidi: embed;}</P>
   <P>english21
      <SPAN class=arabic2>22   english23 24  </SPAN>
   </P>
</DIV>
</BODY>
</HTML>

Notice the difference in the Unicode-bidi assignment given to SPAN.arabic1 and SPAN.arabic2 in the previous code sample. Figure 14-2 illustrates using the correct Unicode-bidi assignment to produce the appropriate layout.

Figure 14.2 Using Unicode-bidi to obtain the appropriate layout.
Figure 14.2 - Using Unicode-bidi to obtain the appropriate layout.


International Best Practices

Since the chapters within Part II, "Globalization," and Part III,"Localizability," provide valuable guidelines for making Web-related software and content world-ready, this information will not be repeated here.


Additional Resources

For more information on HTML, see the following resources:

  1. http://www.microsoft.com/globaldev/articles/mideast.asp. "Authoring HTML for Middle Eastern Content." Gives information on bidirectional text and an overview of the HTML 4 and CSS2 specifications as they pertain to Middle Eastern content.
  2. http://www.w3.org/markup. Provides authoritative information about HTML, including a historical overview and international information.
  3. http://msdn.microsoft.com/workshop/browser/hosting/Hosting.asp . Supplies information about the MSHTML base.
  4. http://www.ietf.org/iesg/1rfc_index.txt . Contains the index of all published RFCs.


Summary

HTML is the basic mechanism by which documents on the Web are presented and linked to each other. Over time, HTML has become a very rich markup language that allows the representation of the most complex layouts. In addition, HTML has many inherent international features that make it a useful tool for dealing with multilingual content. With the emergence of HTML 4, support for bidirectional text was added. Additionally, HTML now has an important dynamic component to it, through the use of scripting languages such as JScript and VBScript.

HTML also offers numerous features that assist in dealing with different character sets and encodings-including the HTML charset declaration, character entity references, numeric entity references, URL encoding, and the LANG attribute, among others. CSS provides further international capabilities to HTML. With CSS, Web site authors can determine and modify the visual appearance of a large set of documents on a Web server-or even across a range of Web servers-and across a multitude of languages to achieve a common identity. Finally, the DIR (direction) attribute is used to indicate the base directionality of an element: left to right or right to left.

Just as the World Wide Web has entirely revolutionized the world, HTML has also made enormous contributions in terms of how documents are presented and viewed around the globe. In the years to come, HTML presumably will continue to evolve, with ever-increasing abilities to deal with different encodings, character sets, fonts, and other issues affecting international computing and the creation of world-ready software.


Chapter 15: Microsoft Internet Information Services (IIS) and Microsoft Active Server Pages (ASP)

This chapter discusses the benefits of using IIS when creating world-ready applications. Included is an overview of what this technology provides, where it is available, as well as what international features, issues, and international best practices are associated with it. Finally, the chapter will supply you with additional resources regarding this technology.


Overview of the Technology

Glossary
  • Component Object Model (COM): A specification that Microsoft developed for building software components that can be assembled into programs, or that add functionality to existing programs running on Microsoft Windows platforms.
  • Network News Transfer Protocol (NNTP): The Internet protocol that governs the transmission of newsgroups.
  • Simple Mail Transfer Protocol (SMTP): A protocol for sending messages from one computer to another on a network; used on the Internet to route e-mail.

IIS has been a component of the Microsoft Windows Server operating system since Microsoft Windows NT 4. It consists of a collection of software services that support Web site creation, configuration, and management, along with other Internet functions. IIS includes such technologies as NNTP and SMTP. Another integral technology is ASP, a server-side scripting environment for creating dynamic, interactive Web server applications. With ASP, you can combine HTML pages, script commands, and calls to COM components that perform a variety of tasks, such as connecting to a database or processing business logic.


Availability

Table 15-1 matches the IIS version to its corresponding operating-system version.

Table 15-1 Availability of IIS.

IIS Version | Operating-System Version
IIS 4 | Microsoft Windows NT 4 Option Pack
IIS 5 | Microsoft Windows 2000
IIS 5.1 | Microsoft Windows XP Professional
IIS 6 | Microsoft Windows .NET Server


International Features

IIS and its related technologies-such as ASP-are key to developing a non-English Web site. Some of the international features associated with IIS include its code-page settings, request collections, Uniform Resource Identifier (URI) processing, and logging-among other features discussed in the sections that follow.

Code-Page Settings

Glossary
  • Global.asa: A file typically containing scripts that initialize application or session variables, connect to databases, send cookies, and perform other operations pertaining to the ASP application or to the user's session with an ASP application as a whole.
  • Indexing Service: A base service of Windows NT, Windows 2000, and Windows XP that extracts content from files and constructs an indexed catalog to facilitate efficient and rapid searching. Indexing Service can extract both text and property information from files on the local host and on remote, networked hosts.

When the server receives a request for an ASP file, it processes server-side scripts contained in the file to build the Web page that is sent to the browser. In IIS 6, ASP and the script engines it supports use Unicode internally. If you author all of your pages in the default code page of the Web server, ASP automatically converts strings to Unicode. (Again, this is true only for IIS 6.) If your script was not created for the Web server's or the browser's default code page, you need to specify the code page. This will allow strings to be correctly converted as they are passed between the script and the ASP engine, between the ASP engine and the browser, and between the ASP engine and COM components. To specify the code page for an ASP page, you can use the Response.CodePage property, the Session.CodePage property, the AspCodePage metabase property, the @CODE-PAGE directive, locale ID (LCID) settings, and the HTTP Charset attribute. (For more information on Response.CodePage and Session.CodePage, see Chapter 3, "Unicode." ) Since IIS does not offer its own code-page or locale support-such as National Language Support (NLS) features-the support available within Windows is used. Consequently, when you select a code page (charset) or locale for an ASP page, session, or response, you must have the appropriate Windows support installed on the server computer.

Response.CodePage Property

When a script is executed, Response.CodePage determines how characters are encoded. If your ASP page runs on IIS 5.1 or later, it is always better to set the value of Response.CodePage explicitly. This way, you eliminate the implicit behaviorthat can cause text transformations you do not expect. If Response.CodePage is not set explicitly in a Web page, then it is set implicitly with this hierarchy:

  • If sessions are enabled and Session.CodePage is set explicitly, Session.CodePage sets Response.CodePage.
  • If @CODEPAGE is defined at the top of the page, @CODEPAGE sets Response.CodePage.
  • If the AspCodePage metabase property for the application is set to a value other than zero, AspCodePage sets Response.CodePage.
  • Otherwise, the Web server's default system code page sets Response.CodePage.

Response.CodePage allows applications that don't enable session state to specify a code page dynamically. For run-time execution of ASP, the following logic for initializing Response.CodePage is used.

If session state is disabled:

if (@CODEPAGE defined)
   Response.CodePage = @CODEPAGE value
else if (AspCodePage property present)
   Response.CodePage = value at AspCodePage
else
   Response.CodePage = CP_ACP
Note
CP_ACP is a server variable that represents the active Windows (sometimes called "ANSI") code page on the user's system.

If session state is enabled:

if (Session.CodePage set explicitly (via script execution))
   Response.CodePage = Session.CodePage
else if (@CODEPAGE defined)
   Response.CodePage = @CODEPAGE value
else if (AspCodePage property present (implicitly sets Session.CodePage))
   Response.CodePage = value at AspCodePage
else
   Response.CodePage = CP_ACP

Session.CodePage Property

If session state is enabled and the value of Response.CodePage is not set explicitly, the value of Session.CodePage is used during script execution to dynamically specify how the strings coming back from the script engine are to be sent to the client.

AspCodePage Metabase Property

Residing at the application level in the metabase, AspCodePage acts as the default value for any of the initial values of Session.CodePage, Response.CodePage, and @CODEPAGE. However, this default value can be overridden by @CODEPAGE when compiling an ASP page. The compile-time code page for Global.asa is specified by using AspCodePage.

@CODEPAGE Directive

This value is used to compile the ASP file, such as in converting the contents of the ASP file to Unicode, as required by the script-engine interface. Specifying a code page in this manner ensures that your literal strings are converted correctly. The value is used for run-time execution, but has no life beyond the executing script. For compiling the ASP page, the order is:

if (@CODEPAGE)
else if (AspCodePage property is present)
else use CP_ACP

However, you should be aware that if Response.CodePage is initialized by Server.Execute using a different value for @CODEPAGE than that of the calling ASP, the encoding is not reset to the calling ASP upon return from the child ASP. Instead, the encoding stays in the new code page.

LCID Settings

The same basic logic that Response.CodePage, Session.CodePage, and @CODEPAGE use applies to the LCID settings Response.LCID, Session.LCID, the AspLCID metabase property, and @LCID. If Session.LCID is not explicitly set in a page, it is implicitly set by the AspLCID metabase property. If AspLCID is not set, or if it is set to zero, Session.LCID is set by the default system locale. Session.LCID can be set multiple times in one Web page and used to format data each time. (For more information on LCIDs, see Chapter 4, "Locale and Cultural Awareness." )

HTTP Charset Attribute

The Charset attribute, which specifies the character set to be used in an HTML page, remains unchanged for IIS 6. The Indexing Service uses the character set specified by Charset to properly display text in the HTML page. It is recommended that you set Response.CodePage, Response.CharSet, and the Charset attribute to the same value. In a file written in Japanese, you would set the Charset attribute to shift_jis as shown in the following example:

<META HTTP-EQUIV="Content-Type" CONTENT="text/html;
   charset=shift_jis">

Another international feature of IIS involves request collections. The following section discusses how these collections work.

Request Collections

Because customers were once obliged to choose non-Unicode character sets instead of UTF-8 to build content on IIS 4 and IIS 5, and because ASP execution on IIS 4 and IIS 5 didn't fully support UTF-8, support for Windows code pages is still necessary. The processing code page used by the request collections is set with the Response.CodePage command. In addition, UTF-8 support has been added for the five types of request collections in ASP: ClientCertificate, Cookies, Form, QueryString, andServerVariables.

ClientCertificate

Windows XP security fully supports ClientCertificate encoded in UTF-8.

Cookies

IIS 6 will convert cookie parameters to and from Unicode using the current Response.CodePage. This affects NAME, VALUE, PATH, and DOMAIN elements in the cookie. If an element of the cookie contains nonalphanumeric characters, it will be URL-encoded with % escaping and hexadecimal codes after WideChar-ToMultiByte conversion.

Form

All properties of the Form collection support Windows code pages and UTF-8.

QueryString

All properties of the QueryString collection support Windows code pages and UTF-8.

ServerVariables

In IIS 6, all file names and metabase paths are in Unicode. This is a change from IIS 4 and IIS 5 where file names and paths were dependent on CP_ACP. In addition, the core Web server maintains them as Unicode, and ASP now flows through into the scripting engines. To provide for these changes, the ServerVariables in Table 15-2 now also support Unicode. (Go to http://msdn.microsoft.com to learn more about each variable.)

Table 15-2 Unicode-supported ServerVariables.
APP_POOL_ID | LOGON_USER
APPL_MD_PATH | PATH_INFO
APPL_PHYSICAL_PATH | PATH_TRANSLATED
AUTH_USER | REMOTE_HOST
CERT_ISSUER | REMOTE_USER
CERT_SERVER_ISSUER | SCRIPT_NAME
CERT_SERVER_SUBJECT | SCRIPT_TRANSLATED
CERT_SUBJECT | SERVER_NAME
HTTPS_SERVER_ISSUER | UNMAPPED_REMOTE_USER
HTTPS_SERVER_SUBJECT | URL
INSTANCE_META_PATH |

URI Processing

Another international feature of IIS 6 is URI processing. ASP can now receive code page-based or UTF-8 URIs from the client. When a client sends a URI that contains characters from the double-byte character set (DBCS), IIS first tries to resolve the URI as UTF-8. If IIS cannot connect to that URI, the URI is assumed to be encoded in the active Windows code page of the server. If it's not a UTF-8 string, MultiByteToWideChar is called to convert the URI string to Unicode. When a client sends a UTF-8 URI, IIS looks at the URI and tries to determine if it is a UTF-8 URI. If it is, the URI is simply converted back to a Unicode string.

On servers where the active code page is a DBCS encoding, the algorithm first tries to resolve the URI as a DBCS string in the current system locale, and not as a UTF-8 string. The rest of the algorithm is not changed.

Other Methods and Components

IIS 6 now supports Windows "ANSI" code-page or UTF-8 parameters for the methods and components shown in Table 15-3. (Go to http://msdn.microsoft.com to learn more about these methods and components.)

Table 15-3 Methods and components supported by UTF-8.
Server.Execute | ContRot.dll
Server.MapPath | NextLink.dll
Server.Transfer | FileSystemObject
Server.URLEncode | Logging Utility
Server-Side Includes | Page Counter Component
Global.asa | Permission Checker Component
AdRot.dll | Tools Component

IIS 6 also supports many new Unicode server-support functions. UNICODE_ server variables are not exposed to ASP. These variables should be used in programs that directly access Internet Server Application Programming Interface (ISAPI) extensions, a topic that is beyond the scope of this book. If these variables are used in an ASP page, an error will be returned. Since ASP developers operate in a Unicode world already, there is no need to support this new functionality.

Because of the need to handle content that is increasingly international, logging has met this challenge. The following section explains the support that logging offers.

Logging

To be more internationally flexible, IIS 6 features the ability to write out log files in UTF-8. By default, this feature is turned off, and files are encoded as the active Windows "ANSI" Code Page (ACP). There is a check box at the server level that tells the system whether to use UTF-8 when logging. Once the Web or FTP services have been started, this switch cannot be changed. To modify its setting, these services must be stopped. The names of Web and FTP services in the list of Windows services appear, respectively, as "World Wide Web Publishing Service" and the "FTP Publishing Service."

It is important to be aware that there are a few limitations associated with IIS 4 and IIS 5 in terms of their international support. The following will show you what these issues are and how to handle them.


Issues with IIS 4 and IIS 5

The internal engine of IIS 4 and IIS 5 is based on Windows code pages, so any items that require processing by the engine will use ACP as a base. As a result, for a Web page to display values as UTF-8, those values will have to be converted from ACP to UTF-8, resulting in some character degradation. Table 15-4 shows the values that are affected.

Table 15-4 Values needing conversion for UTF-8 Web-page usage.
URL | Server.MapPath
PATH_INFO | Server.Transfer
AUTH_USER | Server.URLEncode
LOGON_USER | Server-Side Includes
REMOTE_USER | Global.asa
SCRIPT_NAME | AdRot.dll
APP_POOL_ID | ContRot.dll
APPL_MD_PATH | NextLink.dll
PATH_TRANSLATED | FileSystemObject
SCRIPT_TRANSLATED | Logging Utility
APPL_PHYSICAL_PATH | Page Counter Component
UNMAPPED_REMOTE_USER | Permission Checker Component
Request.Form | Tools Component
Request.QueryString | My Info Component
Server.Execute |

Response.Write, entity data, and query-string data are never processed in the core engine and can use any code page (including UTF-8).


International Best Practices

There are many issues associated with establishing international Web sites and with making content in general applicable to an international audience. Various points to consider-such as the existence of various glyphs, fonts, and character encodings as well as the benefits of using NLS functions for handling locale-specific conventions-have been discussed in the chapters within Part II, "Globalization."


Additional Resources

For more information on IIS, see the following resources:

  1. Appendix H, "Code-Page Support for Microsoft Windows 2000 and Microsoft Windows XP." Reference material that explains code-page support, which IIS uses for encoding-related operations.
  2. Appendix C, "Table of Locales Mapped to Code Pages." Reference information that discusses matching locale IDs to languages and code pages. This matching might be needed, for example, if you want to guarantee that locale-specific data can be displayed in the selected encoding.
  3. http://msdn.microsoft.com/library/default.asp . "Scalable Hosting Solution Overview." Gives an overview of an Internet Server Application Programming Interface (ISAPI) application used in an Internet Service Provider (ISP) hosting environment.
  4. http://msdn.microsoft.com/library/default.asp . "Developing International Database Applications Using MS SQL Server." Provides database developers with a comprehensive guide on building Microsoft SQL Server 6.5 database applications for international use.
  5. http://www.microsoft.com/technet . Ebben, Sjoert, and Gwyneth Marshall. "The Localization Process: Globalizing YourCode and Localizing Your Site." April 1999. Gives an overview of some of the difficulties involved in localization, various steps in the process, and some tips and tricks.
  6. http://www.microsoft.com/technet . Ebben, Sjoert, and Gwyneth Marshall. "Designing a Globalized and Localizable WebSite." August 1999. Discusses the specifics of creating a globalized and localizable Web site. Gives tips about code-page decisions, layered graphics, content mechanisms, and an understanding of how HTML and cascading style sheets (CSS) can be used in globalization.
  7. http://www.microsoft.com/technet. Meade, John. "Using ASP to Internationalize Your Web Pages on Windows 2000." March 2000. Explains how to take advantage of the Windows 2000 and ASP language- support features mentioned earlier-by describing five scenarios.
  8. http://www.microsoft.com/technet . "Developing an International Site." December 2000. Describes best practices for supporting multiple languages and currencies when developing a Web site for an international audience, with the goal of making it easy for your customers to change language or currency at any time without losing context.
  9. http://msdn.microsoft.com/library/default.asp. Pollack, Seth."International Active Server Pages." April 1997. Offers good reference material about how IIS performed in early releases.
  10. http://msdn.microsoft.com/library/default.asp. Clinick, Andrew."It's All Greek to Me." August 1999. Discusses how Microsoft VBScript 5 introduced features to address internationalization.
  11. http://msdn.microsoft.com/library/default.asp. "Charset Property." Talks about matching up charsets to code pages.
  12. http://msdn.microsoft.com/library/default.asp. "Character Set Recognition." Contains information about the character sets supported by Microsoft Internet Explorer 5, including the friendly name, aliases, minimum operating system, and the code page that supports specific character sets.
  13. http://msdn.microsoft.com/library/default.asp. Kaplan, Michael. "Go Global: Designing Your ASP-based Web Site to Support Globalization." July 2000. Examines the Trigeminal Software site, which has pages localized into approximately 48 languages. Contains information on which database to use for storing dynamic content, whether to use static or dynamic pages, and how to implement localized solutions on both Windows NT 4 and Windows 2000.


Summary

IIS and its related technologies-such as ASP-are vital to developing a multilingual Web site and a Web-based application that is world-ready. Some of the international features associated with IIS include its code-page settings and request collections. Within IIS 6, new international features include URI processing and logging.

When the Web server receives a request for an ASP file, it processes server-side scripts contained in the file to build the Web page that is sent to the browser. If your script was not created for the Web server's or the browser's default code page, you need to specify the code page. This will allow strings to be correctly converted as they are passed between the script and the ASP engine, between the ASP engine and the browser, and between the ASP engine and COM components. To specify the code page for an ASP page, you can use the Response.CodePage property, the Session.CodePage property, the AspCodePage metabase property, the @CODEPAGE directive, LCID settings, and the HTTP Charset attribute.

As a caveat, you should be aware that the internal engine of IIS 4 and IIS 5 is based on Windows code pages, so any items that require processing by the engine will use ACP as a base. As a result, for a Web page to display values as UTF-8, those values will have to be converted from ACP to UTF-8, resulting in some character degradation.

There are numerous issues associated with establishing international Web sites and with making content applicable to an international audience, many of which have been discussed previously throughout this book. However, some general guidelines-which extend beyond the scope of IIS-include designing your application for a world-wide market from the beginning, using the correct character encoding, and taking advantage of NLS functions. In combination with the capabilities of IIS, these guidelines will serve you well as you create world-ready applications.


Chapter 16: Microsoft Office

This chapter discusses the benefits of using Microsoft Office when creating world-ready applications. Included is an overview of what this tool provides, where it is available, what international features and international best practices are associated with it, and what alternatives are available. Finally, the chapter will supply you with additional resources regarding this tool.


Overview of the Tool

As personal computers have spread around the world and the Microsoft Windows operating system has gained new international capabilities, Office has followed suit. Because of its close ties to text, Microsoft Word has always had significant support for multiple languages. From Microsoft Office 97 to Microsoft Office XP, this support has expanded dramatically with each new release, thanks to its built-in Unicode support. In addition to Word, other applications such as Microsoft Excel, Microsoft PowerPoint, Microsoft Access, Microsoft FrontPage, Microsoft Publisher, and Microsoft Project have moved to Unicode as well, and all of these products continue to support additional languages with each new release. Moreover, in terms of e-mail, Outlook supports Unicode, along with other national and international encodings in the body of the e-mail, and its developers are in the process of making Outlook completely Unicode-based.

The international support available in Office can help you design and develop an application that is world-ready in two ways. First, because the entire Office suite itself is world-ready, its applications offer an excellent design and development model for your own application. Second, Office is a powerful tool for creating applications if you develop add-ins or other code that calls the programmable object models of each application. (For more information, see "Alternatives" later in this chapter.)

For years, the Office team provided language support for customers in particular markets with the implicit assumption that most people in the world use their native language and perhaps English. Accordingly, localized versions of applications were created that supported the specific target language and market. Some languages-such as those with complex scripts-required special enabling and additional features, including the ability to handle right-to-left (RTL) text, contextual shaping, combining characters, and bidirectional text. As a result, special teams-devoted to enabling Office applications for a specific language and market-created forked versions of applications, whereby each language version contained its own unique source code (which was separate from source code used in other language versions).

With Office 97 this approach of creating a localized version for each target language began to change. Development teams for Word, Excel, and PowerPoint decided to use Unicode to simplify internal development. The Microsoft Office 2000 team strove to globalize the Office suite, making it more attractive to large multinational corporate customers who wanted to deploy the software worldwide and not be concerned with managing different language versions. Support for East Asian languages was merged with support for languages that use complex scripts-such as Arabic and Hebrew-into a single version that supported pluggable language resources.

The Office XP team extended the globalization push of the Office 2000 team by including any additional languages supported by Microsoft Windows 2000 and Microsoft Windows XP. (These languages include Thai, Hindi, Tamil, Armenian, and Georgian, among others.) The following section provides details about the availability of Office.


Availability

Since Office applications are based on Unicode, running them on Microsoft Windows 95, Microsoft Windows 98, and Microsoft Windows Millennium Edition (Me) was initially problematic because these operating systems are based on code pages rather than on Unicode. The Office team was among the first in attempting to solve this issue. Although the original release of Windows 95 had very limited support of Unicode application programming interfaces (APIs)-and frequently had bugs-this support, albeit limited, had a significant impact. With careful testing and coding, it was possible to use those few APIs to create a layer that simulated Microsoft Windows NT. This layer was subsequently used for all Office applications that attempted to output Unicode text.

Accordingly, both Office 97 and Office 2000 were able to run on Windows 95/98/Me, and Office XP supports Windows 98 and later. In addition, Office 97 could output Unicode text to a screen and printer with only a few more limitations than would be experienced if it had been running on Windows 2000. However, text display and printing are only part of the story. When running on any operating system-including Windows XP-Office XP can support concurrent input and processing of all languages that the operating system supports.


International Features

Office provides a host of international capabilities. A complete list of features that have been specifically created or modified for international users is not practical in this space, but some examples include proofing capabilities, support for Asian-language input, sorting support, date and currency formats, support for input/output (I/O) of text files and of Web pages in any encoding, and application-specific features for certain markets.

Proofing in over 40 Languages

Office applications provide multiple spelling-checker engines. They also offer considerable assistance to the international user through a grammar checker, thesaurus, hyphenation options, and bilingual translation dictionaries.

Support for Asian-Language Input

Office applications, especially Word, have always had rich support for features associated with Input Method Editors (IMEs), the components that allow the user to enter the thousands of different characters used in East Asian languages with a standard 101-key keyboard. These features include:

  • Modeless input. A method of entering East Asian language input with an IME. In contrast with modal input, modeless input allows you to easily and seamlessly switch back and forth between the composition mode and the direct mode (the document itself). You can easily correct, navigate, and make input to the general document, whereas with modal input the rest of the document is temporarily unavailable as long as you are in composition mode.
  • Reconversion of errors. A feature that allows you to correct inad-vertent input-conversion errors. For example, the word "kaeru" in Japanese means both "frog" and "go home." If the IME picked the wrong one-and you didn't notice until much later-you can recon-vert the word by right-clicking it (or selecting it), and then using the Convert key on Japanese keyboards. You can also use the Reconvert command in Office applications or the IME toolbar.
  • Mode biasing. Incorporation of logic that biases the method used for input (mode) toward the type of input that is expected. For example, the name field on a Contacts dialog box will have information telling the input method expected for people's names. With this information, the input method biases toward those results and thus provides more accurate input.

These features extend the capabilities of the IMEs to make input of East Asian text easier. (For more general information on IMEs, see Chapter 5, "Text Input, Output, and Display.")

Support for Sorting in a Wide Array of Languages

Collation or sort order, which is the way characters or ideographs are sorted, varies among different languages, countries, and regions. Office handles this task automatically.

Whenever sorting is needed-whether you are creating Access databases or lists in Excel-Office uses the correct sort order.

It is important to note that the meaning of "correct sort order" can change between contexts, which is why each Office application handles collation settings differently. Outlook uses the collation settings of the operating system to determine how to handle sorted lists of contacts. Excel uses the default user locale to determine sort order. Access uses the sort order of a particular database to determine how to return ordered data (a setting that can be changed during the database Compact operation). Word, on the other hand, supports marking specific text in a document as being part of a different language, and thus if you choose to sort data in a table, it can rely on this particular setting rather than on the default setting. (For more information on sort order, see Chapter 4, "Locale and Cultural Awareness." )

Proper Date and Currency Formats

Just as sort order differs from one language, country, or region to another, so do date and currency formats. Once again, Office uses the correct format based on the application and the particular context at hand. For example, the date format in an Excel spreadsheet will follow the preferences indicated in Regional And Language Options within Control Panel, by default. Likewise, the choice of a currency symbol in Regional and Language Options will affect the way that newly added currency fields are formatted in an Access table that is created. Furthermore, applications like Access do not blindly apply the settings from Regional And Language Options in a way that would change the meaning of existing data; because of this, existing data in a table will keep its original currency format, so that $2.00 does not suddenly appear to be £2.00 or ¥2.00. (For more information on date and currency formats, see Chapter 4.)

Support for I/O of Text Files and of Web Pages in Any Encoding

This support includes many types of Unicode, local code pages, standards from the International Organization for Standardization (ISO), national standards, and so on.

Application-Specific Features for Certain Markets

Within the Office product suite, individual applications provide their own international features geared toward particular markets. These features are too numerous to list here, but the following sampling gives you a good idea of these individual applications' versatility. Application-specific features include:

  • A table-drawing tool, which Word pioneered to make creation of complex tables easier. In many Asian countries, typical documents are laid out as forms, with all the content contained in boxes. The table-drawing tool, which is now in Excel and PowerPoint, provides an easy way to create tables with odd shapes.
  • Special text layout for Asian languages, such as phonetic guides, horizontal text in vertical flow, and inline annotations within Word, Publisher, and PowerPoint.
  • Vertical text for Asian languages in many Office applications such as Word, Publisher, and PowerPoint. Bidirectional text support in all applications in the Office suite.
  • Locale-sensitive presentation templates in PowerPoint.
  • Proper local address formats and parsing in Outlook.

You have seen some of the international features that Office provides. By adhering to some basic international guidelines, you can take full advantage of all these capabilities in order to make your application world-ready.


International Best Practices

The following are recommendations when working with Office in the global environment. Concepts that have been developed throughout this book resurface here, such as the importance of having a single binary, of globalizing your application from the start, and of making your application localizable.

Maintain a Single Code Base

In addition to a single code base, try to maintain-as much as possible-a single code path, regardless of the specific Windows version on which you need to run your application. As mentioned earlier, the Office 97 team decided to write code for Windows NT, and to simulate Windows NT on Windows 95/98/Me by using a layer. This practice kept the code simple and encouraged future-and even exclusive-support for for Windows NT-based operating systems such as Windows 2000 and Windows XP.

Consider Your User

The market for Windows 95/98/Me will soon begin shrinking as Windows XP gains usage. Carefully examine where your software sales are. You might assume that you need to support Windows 95/98/Me, but the majority of software is sold to people buying new PCs, or to those who purchased PCs in the last year. As Windows XP becomes more widespread, you might be able to drop support for Windows 95/98/Me sooner than you think. This makes coding your application for globalization significantly easier. If a globalized application is important to your user, which is most likely the case, consider making your application support only Windows NT, Windows 2000, or Windows XP.

Don't Forget to Make Your Application Localizable

Adapting something so that it is localizable is an obvious practice in terms of strings. However, you should also be prepared for when you need to change bitmaps, user interface (UI) direction, or even logic rules (such as the order of user operations) in order to make your application localizable. (For more information, see Chapter 7, "Software Localizability Guidelines," and Chapter 9, "Content Localizability Guidelines.")


Alternatives

Glossary
  • Smart tags: A technology introduced in Office XP that provide users with the ability to associate text and data with actions. Smart tags can provide actions for names, dates, times, telephone numbers, addresses, stock ticker symbols, and so on.
  • Web Parts: Reusable components that contain Web-based content, such as Extensible Markup Language (XML) code, HTML pages, or script. Web Parts support a set of standard properties that determine how the Web Parts are rendered in a Digital Dashboard. Microsoft Office XP Developer makes it possible for you to create and customize Digital Dashboards and Web Parts.
  • Digital Dashboard: A container that provides a customized display of information consolidated from various information sources. Every Digital Dashboard is a Web page containing one or more Web Parts.

Office is itself a software platform. As mentioned earlier, you can harness all the work that the Office development team has done by building your application on top of an Office application, and then by using Component Object Model (COM) interfaces to the object model of Office applications (including Word, Excel, PowerPoint, and Outlook). You can also take advantage of Microsoft Visual Basic for Applications and of extensive technologies such as smart tags, Web Parts, and XML within Office applications. All of these technologies have rich multilingual support built in, so you don't have to worry about providing this.


Additional Resources

For more information on Office, see the following resources:

  1. http://www.microsoft.com/office . Provides general information regarding Office XP.
  2. http://www.microsoft.com/office/ork/xp/three/inta00.htm .Contains the Microsoft Office Resource Kit, as well as information on planning international deployment of Office, upgrading and maintaining international installations, and preparing the user's computer for international use.
  3. http://www.microsoft.com/office/developer . "Office XP for Developers." Gives technical information on how globalized and localizable applications can be written with Microsoft Office.


Summary

From Microsoft Office 97 to Microsoft Office XP, international support has expanded dramatically with each new release of Office because of its built-in Unicode support. Examples of international features that Office provides include proofing capabilities in over 40 languages, support for Asian-language input, sorting support, locale-aware date and currency formats, support for I/O of text files and of Web pages in any encoding, and application-specific features for certain markets. Because of their multilingual support, Office applications are a good design and development model for your own world-ready application. Also, you can take advantage of all the work that the Office development team has done by building your application on top of an Office application, and using COM interfaces to the object model of Office applications.

When working with Office in a global environment, maintain a single code base. Consider the user-as Windows XP becomes more widespread, you might not need to support Windows 95/98/Me. This makes coding your application for global support significantly easier. Design for global support from the start. Also, don't assume that the international user merely needs translated software, or that the software needs to be completely different from that of the original-language version. Finally, make sure your application is localizable.


Chapter 17: MLang

This chapter discusses the benefits of using MLang when creating world-ready applications. Included is an overview of what this technology provides, where it is available, what international features are associated with it, and what alternatives are available. Finally, the chapter will supply you with additional resources regarding this technology.


Overview of the Technology

The multilingual nature of the Internet can cause a wide variety of problems for developers of Internet-related applications. Among these difficulties are detecting the character encoding that Web pages and e-mails use, converting text from one encoding to another as part of an import or export operation, and displaying characters that are not included within the font specified for parts of a page.

MLang (MLang.dll) is a Component Object Model (COM) component that provides a variety of services intended to help solve these problems. Because MLang services are geared toward Internet-related applications, MLang includes application programming interfaces (APIs) to enumerate supported code pages and locales, and to retrieve locale and character-set information.

MLang.dll also includes a rich set of character-set conversion APIs, the most important of which translates Internet character sets to Unicode and back. This is one of the technologies that allows you to base your applications entirely on Unicode and frees you from dependencies on particular Internet character sets.

MLang also provides two code-page detection methods that automatically determine the languages and code pages in which data is written. With MLang you can detect which code pages contain a given character or string.

In addition, MLang gives information about the charset and font support for specific scripts available on the computer that is running the software. This script and font information can be used for implementing font linking, which is a general way to combine several fonts for handling multiple scripts. Through font linking, you can create custom fonts to display characters from a variety of character sets at once.

MLang services supplement the Microsoft Win32 National Language Support (NLS) API by providing additional functionality not offered by NLS. (For more information on NLS, see Chapter 4, "Locale and Cultural Awareness." ) In particular, MLang extends the set of character encodings supported natively by Microsoft Windows operating systems.

Note
In order to use the character-set conversion routines that MLang provides, the NLS files (*.nls) corresponding to the code pages you wish to use can be installed on your system.


Availability

MLang.dll was first distributed with Microsoft Internet Explorer 4 and has since been shipped with every version of Internet Explorer up to and including Microsoft Internet Explorer 6. Table 17-1 shows the various versions of the Windows operating system (and Microsoft Office) along with corresponding releases of Internet Explorer containing MLang.

Table 17-1 Operating-system (and Office) versions with corresponding releases of Internet Explorer.

Operating-System (and Office) Version | Internet Explorer Version
Microsoft Windows 98 | Microsoft Internet Explorer 4.01 Service Pack 1 (SP1)
Microsoft Windows NT 4 Option Pack | Internet Explorer 4.01 SP1
Microsoft Windows NT 4 SP4 | Internet Explorer 4.01 SP1
Microsoft Office 2000 | Microsoft Internet Explorer 5
Microsoft Windows 98 SE (Second Edition) | Internet Explorer 5
Microsoft Windows 98 SE (Second Edition) for Arabic, Hebrew, and Thai | Internet Explorer 4 enabled for complex scripts
Microsoft Windows NT 4 SP5 | Microsoft Internet Explorer 4.01 SP2
Microsoft Windows 2000 | Microsoft Internet Explorer 5.01
Microsoft Windows 2000 SP1 | Microsoft Internet Explorer 5.01 SP1
Microsoft Windows Millennium Edition (Me) | Microsoft Internet Explorer 5.5
Microsoft Office XP | Internet Explorer 5.01 SP1
Microsoft Windows XP | Internet Explorer 6

The MLang.dll distributed with Internet Explorer 4 implemented the original IMultiLanguage interface. Subsequent releases of Internet Explorer have included newer versions of MLang.dll that utilize additional interfaces. The MLang.dll distributed with Internet Explorer 5 implemented the IMultiLanguage2 interface, which updated IMultiLanguage by adding and expanding methods. These changes created new functionality for code-page detection and enhanced the existing functionality for character-set conversion. IMultiLanguage2 also supports the original capabilities of IMultiLanguage, such as character-set conversion and information retrieval from the Multipurpose Internet Mail Extensions (MIME) database. IMultiLanguage3, implemented in the version of MLang that shipped with Internet Explorer 5.5, is a small addition that extends IMultiLanguage2 by adding functionality for outbound text detection. While IMultiLanguage2 replaces some functionality of IMultiLanguage, IMultiLanguage3 just adds two new functions to IMultiLanguage2.

Third-party developers can also use MLang.dll distributed as part of the Microsoft Internet Explorer Administration Kit (IEAK), which is available at http://www.microsoft.com/windows/ieak/downloads/default.asp. As you will see in the sections that follow, software that interacts with Internet data can be made more international with MLang's numerous programming APIs.


International Features

MLang implements several COM objects that offer support for character encoding and text display. The object at the highest level is the MultiLanguage object, which is exposed by the IMultiLanguage, IMultiLanguage2, and IMultiLanguage3 interfaces. IMultiLanguage2, which should be used instead of the older IMultiLanguage, provides access to all of MLang's other interfaces and objects. These include objects for code-page enumeration, locale enumeration, character-set conversion, code-page detection, and font linking, all of which are discussed in the sections that follow.

Code-Page and Locale Enumeration

There are numerous situations in which a program might need to enumerate the list of code pages and locales that the system supports-for example, to enable the user to invoke a preference. However, the code pages and locales that Windows supports can vary from one installation to another.

To help you determine which code pages and locales are supported on a given installation, MLang provides an object for code-page and locale enumeration. These objects can be used to retrieve the code pages and locales that the system recognizes from the MIME database. Enumerating code pages and locales involves the following three steps:

  1. Obtain a pointer to the IMultiLanguage2 interface.
  2. Call either the IMultiLanguage2::EnumCodePages method or the IMulti-Language::EnumRfc1766 method to obtain an interface to the appropriate enumeration object.
  3. Call the Next method of the enumeration interface that is obtained.

When enumerating code pages, the appropriate MIMECONTF constant should be used as the grfFlags parameter for the EnumCodePages method call. The MIMECONTF constants specify a subset of code pages (such as code pages that mail, news, or browser clients use). Additionally, these constants allow you to tailor the list of code pages returned.

Another international feature of MLang is its support for character-set conversion. The following shows the interfaces and objects that you can use.

Character-Set Conversion

MLang provides a number of different ways to convert strings from one code page to another, the best of which is to use the MLang Conversion object. This object supports the IMLangConvertCharset interface and is dedicated to converting strings between a source code page and a destination code page, both of which the user must specify.

The IMultiLanguage2 interface also supports a number of conversion methods. These methods function in the same manner as the IMLangConvertCharset methods, but they take slightly different parameters. The IMultiLanguage2 methods are less efficient than IMLangConvertCharset methods when doing multiple conversions between the same combination of source and destination code pages.

When using MLang's conversion functionality, a caller should always try to verify whether the parameters that specify the size of the source string and destination buffer are measured in a byte count or character count. In general, all APIs that are specifically dedicated to converting to or from Unicode take a character count for the Unicode string. All other strings are measured in a byte count.

MLang provides several conversion methods, which include methods associated with IMLangConvertCharset and IMultiLanguage2.

IMLangConvertCharset methods:

  • IMLangConvertCharset::DoConversion
  • IMLangConvertCharset::DoConversionFromUnicode
  • IMLangConvertCharset::DoConversionToUnicode

IMultiLanguage2 methods:

  • IMultiLanguage::ConvertString
  • IMultiLanguage::ConvertStringFromUnicode
  • IMultiLanguage::ConvertStringToUnicode
  • IMultiLanguage2::ConvertStringFromUnicodeEx
  • IMultiLanguage2::ConvertStringToUnicodeEx
  • IMultiLanguage2::ConvertStringInIStream
  • IMultiLanguage::IsConvertible

Another issue that sometimes arises when working with Internet-related applications is the inability to tell which code page was used for a file. The following discusses how MLang can be used to detect code pages.

Code-Page Detection

Authors often neglect to mark the code page or language of a Web page or other file on the Internet. This can cause problems for programs that need to convert text from the input code page or language into something else. MLang provides two code-page detection methods for determining the possible code pages and languages of the text data that is given by the caller:

  • IMultiLanguage2::DetectInputCodepage
  • IMultiLanguage2::DetectCodepageInIStream

These methods return the results in an array of DetectEncodingInfo structures. In addition to containing a detected language and code page, these structures include two members that indicate the percentage of the data that is in the detected language, as well as the relative confidence that the structure contains the correct language and code page. To help increase the accuracy of the detected language and code page, the MLDETECTCP enumerated type is provided. The values associated with MLDETECTCP specify the type of data that is being given to the detection method.

The numerous fonts that exist in multilingual content present special challenges. Individual character sets have their own corresponding fonts. Once again, MLang provides an effective solution, in this case via font linking.

Font Linking

Through font linking, MLang allows you to output text containing characters from a number of different languages and character sets that the specified font might not support. (For more information on fonts and on font linking, see Chapter 5, "Text Input, Output, and Display.") This functionality is especially useful when dealing with Unicode strings, which might contain characters from many character sets at once. Font linking involves the creation of a custom font inherited from a specified source font. The methods used to perform font linking are contained in the IMLangFontLink interface.

Determining the need for-and creating-a custom font is a five-step process:

  1. Use the IMLangFontLink::GetFontCodePages method to return a set of code pages that the specified font supports.
  2. Use the IMLangCodePages::GetStrCodePages (or IMLangCodePages::-GetCharCodePages for a single character) method to return the code pages necessary for the given string.
  3. Do a bitwise comparison of the results of calls 1 and 2.
  4. Use the IMLangFontLink::MapFont method to create a font if the results of the comparison show that it is needed.
  5. Call IMLangFontLink::ReleaseFont to delete the font from the font cache when it is no longer needed.

These steps are illustrated in the following code sample:

DWORD dwFontCodePages;
DWORD dwCharCodePages;
// Get the code page (or pages) supported by the font.
pMLangFontLink->GetFontCodePages(hDC, hFont, &dwFontCodePages);

// Get the code page (or pages) that support the character.
pMLangCodePages->GetCharCodePages(ch, &dwCharCodePages);

// Check to see if the character is supported by the font.
if (dwFontCodePages & dwCharCodePages)
{
   // The character ch can be output by hFont on hDC,
   //    no work needs to be done!
}
else
{
    HFONT hMappedFont;
    pMLangFontLink->MapFont(hDC, dwCharCodePages, hFont,
                       &hMappedFont);
    // The character ch can be output by hMappedFont on hDC.
}


Alternatives

If the services that MLang provides are important to you, but you don't want to create a dependency on an external component, your only other option is to implement the required services yourself. Of course, there are many disadvantages to this. Among other things, you will need to carry around hard-coded lists of encodings, language tables for HTML encodings, locale ID (LCID) and Request for Comments (RFC) 1766 mappings, and font tables. This-combined with the fact that you will need to keep updating your implementation to support such things as new encodings-might persuade you to integrate MLang API calls into your code.


Additional Resources

For more information on MLang, see the following resources:

  1. http://msdn.microsoft.com/workshop/misc/mlang/overview/overview.asp. "Introduction to MLang." Provides information on the two major Internet protocols that MLang uses (MIME and RFC 1766). Also discusses MLang dependencies, the Component Object Library, MLang architecture, and the MIME database.
  2. http://msdn.microsoft.com/workshop/misc/mlang/mlang.asp. "MLang." Gives links to further overviews and tutorials, such as how to break text according to the standards of a particular locale, and how to enum-erate code pages and locales. Also provides information about objects, interfaces, functions, and structures.


Summary

If you are creating an Internet-related application, there are certain character-encoding and text-display challenges that you will likely encounter along the way because of the multilingual nature of the Internet. Fortunately, MLang implements various COM objects that offer solutions for these challenges. The object at the highest level is the MultiLanguage object, which is exposed by the IMultiLan-guage, IMultiLanguage2, and IMultiLanguage3 interfaces.

Specifically, MLang's APIs allow you to enumerate supported code pages and locales, to detect the character encoding and code pages that Web pages and e-mails use, and to convert text from one encoding and character set to another as part of an import or export operation. Finally, you can display characters that are not included within the font specified for parts of a page, through MLang's use of font linking. In short, MLang provides an effective and viable solution for handling the many international and multilingual character-encoding and display issues that are intrinsic to developing an Internet-related application.


Chapter 18: Microsoft Layer for Unicode (MSLU)

This chapter discusses the benefits of using the Microsoft Layer for Unicode on Windows 95, Windows 98, and Windows Me-the full name for what is commonly referred to as "MSLU"-when creating world-ready applications. Included is an overview of what this tool provides and where it is available. Finally, the chapter will supply you with additional resources regarding this tool.


Overview of the Tool

Glossary
  • Common controls: Within the context of Microsoft Windows, a set of controlling elements (windows) that are implemented by the common control library (Comctl32.dll), which is included with the Microsoft Windows operating system. Like other control windows, a common control is a child window that an application uses in conjunction with another window to perform input/output (I/O) tasks.

Back in the early 1990s, two different operating systems were in development at Microsoft. The new platform in development at that time, which was first released in 1993 as Microsoft Windows NT 3.1, was based on the Unicode system of encoding. However, Microsoft Windows 95 to use the Windows code-page model. (For more information on Unicode and code pages, see Chapter 3, "Unicode." )

As a result, application programming interfaces (APIs) that take a character or string parameter behave differently depending on the platform. On Microsoft Windows NT/2000/XP1 these APIs have two versions: Unicode and Windows code pages. The APIs that are based on code pages convert all strings to Unicode, call the Unicode version of the API, and then convert all strings back to the Windows code page associated with the system locale.2 The advantage of using Unicode is that only one version (that is, a single binary) of a Unicode application is needed in order to support multiple languages and scripts.

On Microsoft Windows 95/98/Millennium Edition (Me), there are a few APIs within the system that have both a Windows code-page version and a Unicode version, but all other APIs are exclusively based on code pages. As a result, a Unicode application will not run on Windows 95/98/Me unless the application developer writes a conversion layer between Unicode and the appropriate Windows code page for any system APIs that are used. In addition, because of other text-processing requirements particular to a writing system and its corresponding code page, multiple versions of an an application based on code pages had to be released for different languages (such as for languages in Western, Central, and Eastern Europe, as well as East Asia, and the Middle East).

Initially, it was assumed that developers would circumvent these problems by migrating to Windows NT/2000/XP rather than continue developing products for the Windows 95/98/Me platforms. However, it gradually became apparent that developers were writing applications for Windows NT/2000/XP using Unicode, and then writing a Windows code-page version for Windows 95/98/Me. With MSLU, developers do not need to worry about writing two different versions, but can concentrate on creating a single binary that works on both platforms.

MSLU is a generic conversion layer developed in response to an increasing need for globalized applications that worked just as well on code page-based Windows 95/98/Me systems as they did on Unicode-based NT/2000/XP systems. In other words, this tool bridges the development gap between Windows NT/2000/XP and Windows 95/98/Me that exists by virtue of the difference between the Unicode and code-page model. MSLU is intended for use with any language version of Windows 95/98/Me. It is not needed for Windows NT/2000/XP, since there is no reason to translate Unicode APIs to Windows code-page APIs on these Unicode-based systems. Since Unicode facilitates global development much more than a code page-based model ever can, the future of globalized software rests with Unicode. (For more information, see Chapter 3.) This idea is reinforced by the comprehensive international support built into Windows NT/2000/XP platforms. However, as long as developers need to write applications for Windows 95/98/Me, there has to be an easy way to translate between Unicode functions and functions that are based on code pages. MSLU provides an effective solution.

Nonetheless, MSLU is not a panacea for creating well-globalized applications that run just as well on Windows 95/98/Me as on Windows NT/2000/XP. The architectural challenges inherent in creating a multilingual application on Windows code page-based platforms still exist. First, there is considerably less multilingual support available on Windows 95/98/Me than on Windows NT/2000/XP3 even after implementing MSLU (such as when entering Arabic on Korean versions of Windows 98). Second, MSLU does not make these code page-based platforms support Unicode internally. Third, MSLU was not designed to support existing solutions and components for code page-based platforms that provide their own Unicode functionality, such as MLang (see Chapter 17, "MLang"), Uniscribe (see Chapter 24, "Uniscribe"), Rich Edit (see Chapter 21, "Rich Edit"), common controls, and others. Fourth, you must still follow best practices associated with globalization. (For more information, see the chapters within Part II, "Globalization.") Finally, from an international standpoint, MSLU has a distinct disadvantage when compared to a Unicode platform. While MSLU does provide a Unicode "face" to your applications, it must talk internally to a non-Unicode operating system. Therefore, when your Unicode application that is running on Windows 95/98/Me is limited to a single code page, it is important to make sure that the application behaves properly. A good Unicode application would not be expected to have too many problems. However, you should test your application on each of these three platforms to make sure that nothing breaks and that the user does not lose any data. By first seeing what happens if your application is used beyond what the system can support, you can address any problems before the application is shipped.

Despite these limitations, if you want to ensure that Unicode APIs will be correctly converted for use on Windows 95/98/Me without having to write your own conversion layer, MSLU will fit that need very well. Of course, you can always create your own conversion layer between Unicode and Windows code-page APIs. (For more information, see F. Avery Bishop's article entitled "Design a Single Unicode App that Runs on Both Windows 98 and Windows 2000" at http://www.microsoft.com/msj/0499/multilangUnicode/multilangunicode.htm.) However, one thing that MSLU provides that other layers cannot (or do not) is the conversion of messages sent between different Windows operating systems. Not even the Unicode layer written for Office (as described in Chapter 16, "Microsoft Office") could avoid writing special case code for WM_CHAR or other messages sent between Windows NT/2000/XP and Windows 95/98/Me.

The previous section introduced you to what MSLU is and the advantages it provides. The following sections discuss the elements that comprise MSLU and show how MSLU actually works. They will also demonstrate how to override the loading of MSLU as well as how to override an individual API.

How MSLU Works

MSLU actually consists of two parts. The first part is a dynamic-link library (DLL)-Unicows.dll-that is redistributed with the application that the user creates. The second part is a library (LIB) file-Unicows.lib-that is compiled with this same application. This LIB contains the MSLU loader, which determines whether to load the DLL and call it; alternatively, it can call the original operating-system version of the API. (The former scenario would occur on Windows 95/98/Me; the latter would occur on Windows NT/2000/XP.) The loader also contains all of the necessary failure stubs like the operating system itself does, so that in case the DLL cannot be found, you can gracefully handle the failure of the APIs in your code.

From a technical perspective, the only time your code would ever call the functions in Unicows.dll is when you are using Windows 95/98/Me. Even if you circumvent the loader (for instance, by calling LoadLibrary/GetProcAddress yourself, or by using the P/Invoke syntax of the Microsoft .NET Framework) and call the DLL directly, on an NT/2000/XP platform the DLL will simply forward the call to the API rather than call the MLSU "wrapper" function. This is because the wrapper has less functionality than the full operating-system version, which has full Unicode support. If you are using C or C++, it is not a good idea to intentionally slow yourself down by adding a DLL call you do not need that will simply call the operating system anyway. You can instead decide whether to call the operating-system version of a function or the MSLU version yourself, thus avoiding the performance hit.

With every Unicode wrapper function in MSLU, there must obviously be a conversion to a non-Unicode string. This is usually done using the system default code page (CP_ACP) of the computer-a setting that cannot be changed on Windows 95/98/Me. There are some specific APIs that do not use CP_ACP because they have information in another parameter, such as a locale ID (LCID) value or a device context that would suggest a better code page to use. (For more information, see the MSDN Magazine article listed under "Additional Resources" later in this chapter.)

How (and When) to Handle Overrides

For the most typical uses of MSLU, there is no need to write any code, with two exceptions:

  • When you are overriding the loading of MSLU
  • When you are overriding an individual API
Note
Both of these cases assume you are using the MSLU loader.

If you are using Microsoft C# or Microsoft Visual Basic .NET's P/Invoke mechanism instead of the MSLU loader, you can call the APIs in Unicows.dll directly, and the code in the DLL will determine whether to handle the function itself or to call the operating-system version.

Overriding the Loading of MSLU

There are two steps required to override the loading of MSLU:

  1. Set the "hook" that informs the MSLU loader which function it should call.
  2. Provide the function that the loader will call when it first tries to load the DLL.

The callback function shown in the following code attempts to load the DLL without a path first. This way, if any other component in the process has already loaded the DLL, you can be sure to share the same instance. If the function still fails to find the DLL, it will cause the process to exit. This is the same logic that the loader will use even without an override. Ideal usage of the override would build a path from information in the registry or from some other application-defined location. Exiting is important for a Microsoft Foundation Class (MFC) application, since MFC does not handle the failure of Unicode APIs very well. MFC assumes that certain functions will work. For this reason, an MFC application will crash when trying to dereference a NULL pointer during its initialization-after a call to an API fails-since it does not check for failure in this case.

extern FARPROC _PfnLoadUnicows = (FARPROC) &LoadUnicowsProc;
HMODULE LoadUnicowsProc(void)
{
   HMODULE hMod = LoadLibraryA("unicows.dll");
   if(hMod == 0)
   {
      // Replace <special dir> with your specific path.
      hMod = LoadLibraryA("<special dir>\\unicows.dll");
   }
   if(hMod == 0)
   {
      // If the load still failed, then exit.
      MessageBoxA(0,
         "Unicode wrapper not found",
         "My Company",
         MB_ICONSTOP | MB_OK);
      _exit(-1);
   }
   return(hMod);
}

You must not call any Unicode APIs from the callback function, since the very first Unicode API that is loaded causes the loader to call this function. If you cause another Unicode API to be called, it will try to call itself again, recursively, until it runs out of stack space. Thus no APIs that MSLU wraps should ever be called from this function.

Overriding an Individual API

The syntax for overriding an API is similar to that for overriding the DLL's load. You simply set a hook to inform the loader of which function it should call, and then provide the actual function. The Microsoft Windows Platform SDK documentation, available at http://msdn.microsoft.com, includes a code sample that overrides the LoadCursorW API. However, the sample here provides an override function for an API that is covered by MSLU only in stub form. This API is OleUIInsertObjectW, which calls the InsertObject method that many applications use to support Microsoft ActiveX object insertion.

static UINT __stdcall
MyOleUIInsertObjectW (LPOLEUIINSERTOBJECTW lpouiiow)
{
   UINT result = OLEUI_CANCEL;
   OLEUIINSERTOBJECTA ouiioa;
   memcpy(&ouiioa, lpouiiow, sizeof (OLEUIINSERTOBJECTA));
   ouiioa.lpszFile = (char *)alloca (ouiioa.cchFile + 1);   ouiioa.lpszFile [0] = '\0';
   result = OleUIInsertObjectA(&ouiioa);
   if (result == OLEUI_SUCCESS)
   {
      memcpy(lpouiiow,
         &ouiioa,
         sizeof(OLEUIINSERTOBJECTW));
      MultiByteToWideChar(CP_ACP,
         0,
         ouiioa.lpSzFile,
         ouiioa.cchFile,
         lpowiiow->lpszFile,
         lpowiiow->cchFile);
   }
   return result;
}
extern "C" FARPROC Unicows_OleUIInsertObjectW =
   (FARPROC)&MyOleUIInsertObjectW;

Note that the preceding override will only be called on Windows 95/98/Me.

One important limitation of this override is that it is a part of the MSLU loader, so it is only available in C and C++. However, within those languages, you can use the syntax described here to override as many APIs you want. In fact, you could override every API and simply use the MSLU loader around your own application. You could even call the MSLU API after some special processing by making calls to LoadLibrary ("unicows.dll") followed by GetProcAddress- ("<your function>").


Availability

There are two parts to MSLU: the loader and the DLL itself. The loader is only needed for C and C++ applications. If you are using some other development language, you do not need to download this. You can obtain the loader (Unicows.lib) from the Microsoft Windows Platform SDK site at http://www.microsoft.com/msdownload/platformsdk/sdkupdate/ as part of the "Core SDK."

The DLL is part of the redistribution pack, available from the Windows Platform SDK site at http://www.microsoft.com/msdownload/platformsdk/sdkupdate/psdkredist.htm. Look for MSLU on the list of redistributable packs.


Additional Resources

For more information on Unicode, see the following resources:

  1. http://www.microsoft.com/globaldev/Articles/mslu_announce.asp . "The Microsoft Layer for Unicode on Windows 95/98/Me Systems." Gives a basic introduction to MSLU and provides links to download MSLU.
  2. http://msdn.microsoft.com/msdnmag/issues/01/10/MSLU/MSLU.asp . "MSLU: Develop Unicode Applications for Windows 9.x Platforms with the Microsoft Layer for Unicode." October 2001. Covers MSLU in great detail. Specific implementation information on MSLU is available on the Microsoft Developer Network (MSDN) as well as in the Windows Platform SDK documentation, available from http://msdn.microsoft.com.
  3. nntp://microsoft.public.platformsdk.mslayerforunicode. Provides a dedicated MSLU newsgroup that answers development questions.


Summary

MSLU is a conversion layer developed in response to an increasing need for globalized applications that work just as well on code page-based Windows 95/98/Me systems as they do on Unicode-based NT/2000/XP systems. In other words, with MSLU you can write a single Unicode version of your application and have it run properly on all platforms. MSLU is not needed for applications that will only run on Windows NT 3.1 through Windows NT 4, Microsoft Windows 2000, or Windows XP, since there is no reason to translate Unicode APIs to Windows code-page APIs on these Unicode-based systems.

The distinct benefits of using MSLU when creating world-ready applications far outweigh the few limitations. Since Unicode facilitates global application development much more than code pages, MSLU allows you to leverage Unicode's superiority even on operating systems that are not Unicode-based. Additionally, instead of having to write both a Unicode and a Windows code-page version of an application, you create a single binary that works on both platforms.

It is recommended that you test how your application behaves on Windows 95/98/Me. This allows you to see what happens if your application is used beyond what the system can support, so that you can fix any problems before the application is shipped to the user. Although you can create your own conversion layer between Unicode and Windows code-page APIs, it is much easier to use MSLU, with its thousands of lines of code that already handle many special cases that you might not even consider.


Chapter 19: The Microsoft .NET Framework

This chapter discusses the benefits of using the .NET Framework when creating world-ready applications. Included is an overview of what this platform provides, where it is available, and what international features are associated with it. Finally, the chapter will supply you with additional resources regarding the .NET Framework.


Overview of the Platform

Glossary
  • Remoting: In the .NET Framework, remoting allows objects to interact with one another across application domains. The framework provides a number of services, including activation and lifetime support, as well as communication channels responsible for transporting messages to and from remote applications.

The .NET Framework is a new computing platform that simplifies application development in the highly distributed environment of the Internet. The .NET Framework is designed to accomplish the following objectives:

  • To give the developer a consistent experience across a wide spectrum of applications, such as Microsoft Windows-based and Web-based applications.
  • To help provide a code-execution environment that minimizes software issues and conflicts and that guarantees safe execution of code, including code created by a nonsecure third party.
  • To provide a code-execution environment that does not suffer the performance problems of scripted or interpreted environments.
  • To supply a consistent object-oriented programming environment whether object code is stored and executed locally, executed locally but Internet-distributed, or executed remotely.

The .NET Framework has two main components: the Common Language Runtime (CLR) and the .NET Framework class library. The first component, the CLR, is the foundation of the .NET Framework. You can think of CLR as an agent that manages code at execution time, providing core services such as memory management, thread management, and remoting, while also helping to enforce strict safety and accuracy of the code. In fact, the concept of code management is a fundamental principle of CLR. Code that targets CLR is known as "managed code," while code that does not is known as "unmanaged code."

The other main component of the .NET Framework, the class library, is a comprehensive, object-oriented collection of reusable types that you can employ to develop applications. These applications range from traditional command-line or graphical user interface (GUI) applications to those that are based on the latest innovations provided by Microsoft Active Server Pages for the .NET Framework (ASP.NET), such as Web Forms. Other applications include Extensible Markup Language (XML) Web services, Microsoft ActiveX Data Objects for the .NET Frame-work (ADO.NET), and Microsoft Windows Forms. Figure 19-1 illustrates the basic architecture of the .NET Framework.

Figure 19.1 NET Framework architecture.
Figure 19.1 - NET Framework architecture.


Availability

The .NET Framework SDK is available at http://www.microsoft.com/net.


International Features

The .NET Framework and the invaluable international support it offers are topics that have been discussed extensively throughout Part II, "Globalization," and Part III,"Localizability." For that reason, that information will not be repeated here. The .NET Framework class library provides classes and methods that make it easier for you to build world-ready applications. Within the .NET Framework class library, the most significant namespaces for internationalization are shown in Table 19-1.

Table 19-1 Namespaces within the .NET Framework class library, used for internationalization.
Namespace | Description
System.Resources | Includes information on retrieving translated resources by culture.
System.Globalization | Contains culture-related information, including calendars, format patterns for dates, currency, and numbers, and sort order for strings.
System.Text | Includes encodings such as UTF-8, UTF-16, and Windows code pages.
System.DateTime | Includes DateTime.ToString for conversion of dates to strings.
System.Drawing | Includes GDI+ basic graphics functionality. More advanced functionality is available inSystem.Drawing.Drawing2D, System.Draw-ing. Imaging, and System.Drawing.Text.

All of these classes and methods are explained in more detail throughout Part II, "Globalization."


Additional Resources

For more information about the .NET Framework, see the following resources:

  1. http://www.microsoft.com/net. Introduces the .NET technology, furnishes product information, as well as details on how to use and develop applications with .NET. Also provides downloads, such as for the Microsoft .NET Framework SDK. After downloading the SDK, see the documentation section entitled "Developing World-Ready Applications" for additional information.
  2. http://msdn.microsoft.com/net. Contains a wealth of information on development within the .NET Framework.
  3. http://www.gotdotnet.com. Has tutorials on the .NET Framework, along with downloads, tools, and information on training and events. Also provides links to .NET publications and .NET user groups.
  4. http://v4.windowsupdate.microsoft.com/en/default.asp. Gives the latest updates on releases of Windows products and of products that run on Windows.


Summary

The .NET Framework is a new computing platform that simplifies application development in the highly distributed environment of the Internet. There are countless advantages to using the .NET Framework when creating world-ready applications, just one of which is the development consistency that it enables across a wide spectrum of applications.

The .NET Framework consists of the CLR and the .NET Framework class library. While the CLR is the foundation of the .NET Framework, the .NET Frame-work class library is a comprehensive, object-oriented collection of reusable types for developing applications-from traditional command-line or GUI applications to Web Forms and XML Web services. The .NET Framework class library provides various classes and methods that make it easier for you to build world-ready applications, which include System.Resources,System.Globalization, System.Text, System.DateTime, and System.Drawing.


Chapter 20: OpenType Fonts

This chapter discusses the benefits of using OpenType fonts when creating world-ready applications. Included is an overview of what this technology provides, where it is available, what international features, issues, and international best practices are associated with it, as well as what alternatives are available. Finally, the chapter will supply you with additional resources regarding this technology.


Overview of the Technology

Glossary
  • Outline: A series of contours made up of straight lines and curves that define the shape of a glyph.

Microsoft and Adobe jointly developed the OpenType font format, which was first released in April 1997. It is essentially an extension of the TrueType format-a digital font technology (designed by Apple Computer and now used by both Apple and Microsoft in their operating systems) that offers superior display quality on computer screens and printers. Expanding upon the capabilities of TrueType, OpenType provides better support for international character sets, protection for font data, and advanced typographic control. OpenType fonts can contain either TrueType outlines or PostScript outlines.

One of the advantages of fonts created with the OpenType format is that, by using Unicode, they can handle a large and varied glyph set. Because of the powerful architecture that the OpenType Layout model provides, these fonts also allow a rich mapping between characters and glyphs, thus enabling support for ligatures, positional forms, alternates, and other substitutions.

Another benefit of OpenType fonts is their ability to contain layout tables that give font designers greater typographic control. (For more information, see "International Features" later in this chapter.) Digital signatures can be included in OpenType fonts, which allow operating systems and applications to establish the integrity of the font, identify its source, and find out if it contains any embedding restrictions. Restrictions that the font designer embeds in a font that is digitally signed cannot be altered, which helps ensure the protection of your font data.

Yet another advantage resides in the fact that OpenType fonts are portable across platforms. They can also be embedded in documents or used for international content on Web sites. In addition, the OpenType font format supports all encoding schemes in common use today and, therefore, is not dependent on any single encoding scheme. However, all fonts that Microsoft currently ships are Unicode-encoded.


Availability

Core fonts provided with Middle East and East Asian (previously referred to as the "Far East") versions of Windows 95 were among the first OpenType fonts that Microsoft provided. Since then, the number of OpenType fonts that are shipped, the scripts they support, and the OpenType Layout features they implement have been steadily increasing.

Today, OpenType fonts ship with a wide range of Microsoft products, including Microsoft Office, Microsoft Internet Explorer, Microsoft Windows Millennium Edition (Me), and Microsoft Windows NT, Windows 2000, and Windows XP releases. (For a complete list of fonts and the products they ship with, and for information on specific fonts, go to the Microsoft Typography site at http://www.microsoft.com/typography/fonts/default.asp.)


International Features

Glossary
  • Mark: In typography, a glyph for a character like a diacritic or tone mark that combines with other marks or characters. Marks can be spacing or nonspacing glyphs in a font.
  • Cursive attachment: Used when adjacent glyphs need to be positioned in order to join them cursively. It is heavily utilized in fonts that support cursive scripts like Arabic.
  • Mark attachment: Used in typography when marks need to be attached or positioned to other marks, base glyphs, or ligatures.

While TrueType worked well when developing fonts for languages that use a Latin script, more flexibility was required in order to design fonts for languages that use complex scripts like Arabic, Indic languages, and so on. OpenType Layout provides the necessary infrastructure in fonts for laying out text in multilingual and non-Latin environments, by providing the necessary layout information within five tables in OpenType fonts. (See Table 20-1.)

Table 20-1 OpenType Layout tables.
Table | Description
Baseline table (BASE) | Makes information available about baseline offsets when composing multilingual text in a line
Glyph Definition table (GDEF) | Contains information about the type of glyphs (simple, ligature, mark) used in the GSUB and GPOS tables
Glyph Positioning table (GPOS) | Contains information about x and y positioning of glyphs; positioning can be achieved using single glyph adjustment, pair adjustment, cursive attachments, mark attachments, or contextual positioning
Glyph Substitution table (GSUB) | Contains information about a number of glyph substitution types such as single glyph substitution, substitution of alternate forms of glyphs, ligature substitutions, one-to-many substitutions (decomposing a glyph), and contextual substitutions
Justification table (JSTF) | Contains justification information for glyphs, as well as white-space adjustments and kashidas (continuous horizontal lines inserted between adjoining characters that make a word occupy more space for text- justification purposes)

Layout data in the GSUB and GPOS tables is organized in a well-defined structure consisting of script, language system, typographic feature, and lookup. Scripts, which are defined at the top level, help identify the collection of glyphs in the font that will be used to represent one or more languages. Language systems, which are defined within scripts, can modify the function or appearance of glyphs in a script for a particular language. Language systems define features, which are essentially rules for using glyphs to represent that language. Features are classified based on function into the following categories: those that influence metric behavior, those that address linguistic requirements, and those that allow typographic enhancements. Features, in turn, are implemented by using data in lookups. This data provides information about glyphs that are affected by an operation, the type of operation (substitution or positioning), and the resulting glyph output. The data is used by a text-processing client to substitute or position glyphs. Figure 20-1 depicts the structure of layout data in GSUB and GPOS tables.

Figure 20.1 Structure of layout data in GSUB and GPOS tables.
Figure 20.1 - Structure of layout data in GSUB and GPOS tables.

Scripts, language systems, and layout features are identified using tags made of 4-byte character strings. Each tag has a specific meaning and conveys precise information to developers and text-processing applications. Clients or text-processing applications retrieve and parse information in the GPOS and GSUB tables and then use it to lay out text.

There are times when a glyph needs to be substituted with an alternative glyph depending on the particular context involved. As mentioned, the GSUB table contains information on glyph substitution types, alternate forms, and so on. The following section examines a lookup from a GSUB table that is used for glyph substitution.

Substitution of Alternative Glyph Forms

The following code, from a GSUB table in a Devanagari font, is a lookup that substitutes the glyph sequence hiRaFull hiHalant with hiReph. The glyph hiRa-Full contains outlines for the consonant "Ra" and is mapped to U+0930. The hiHalant glyph contains outlines for Halant (or Virama), and is mapped to U+094D. The Halant (or Virama) is a vowel omission sign whose purpose is to cancel the inherent vowel of the consonant to which it is applied. The hiReph glyph contains outlines for the Reph-an orthographic abbreviation for displaying the sequence of "Ra" Halant (Virama) when a syllable in Devanagari begins with these characters. The substitution of the Reph is a linguistic feature that is required in this script.

Lookup LookupReph
4       ; Lookup Type = ligature subst
0       ; LookupFlag = 0
1       ; Number of subtables
SubstTableReph     ;       Offset to subtable

LigatureSubstFormat1 SubstTableReph
1       ;  Format
CoverageReph    ; Offset to Coverage
1       ;  Ligature Set Count
LigSetRephRaFull

LigatureSet LigSetRephRaFull
1
LigReph

Ligature LigReph
hiReph
2
hiHalant

CoverageFormat1 CoverageReph
1            ; Coverage Format
1            ; Number of glyphs in coverage
hiRaFull

Figure 20-2 illustrates the substitution rule shown in the previous lookup code.

Figure 20.2 Substitution rule in the GSUB lookup code.
Figure 20.2 - Substitution rule in the GSUB lookup code.

Up until now, you have seen some of the advantages that OpenType provides, as well as how OpenType is able to handle layout, substitutions, and so on. However, what enables OpenType to offer capabilities such as the shaping and positioning of glyph strings or the reordering of characters? The following section explores this question.

How OpenType Works

OpenType fonts for international complex scripts make use of Windows operating-system services like Uniscribe (also known as the "Unicode Script Processor") to shape and then position glyph strings that character strings are mapped to. Uniscribe, in turn, uses the OpenType Layout services library, which is a set of helper functions. These functions retrieve glyph substitution and positioning data in a font and lay out the glyph string during text processing.

Uniscribe is a collection of application programming interfaces (APIs) that enables a text-layout client to format complex scripts. Within Uniscribe are multiple shaping engines that contain layout knowledge for various scripts such as Arabic, Devanagari, Hebrew, Thai, among others, some of which require char acters to be reordered within character "clusters." Uniscribe creates and manages a buffer of appropriately reordered character codes. It then obtains the corresponding glyph string by passing the reordered character string to the glyph substitution function of the OpenText Layout services. Because glyph strings are obtained from reordered character strings, the layout features in such fonts are encoded to map reordered characters (and combinations of characters) to their corresponding glyphs. Consequently, font developers are saved from several layers of complexity when they define features, which allows Uniscribe to perform standard character-reordering operations. If a glyph string needs to be both substituted and positioned, substitution is always done prior to any positioning operation. Figure 20-3 illustrates the processing of a sample complex-script string and shows how the substitution rule is applied.

Figure 20.3 Using the substitution lookup in a Devanagari string.
Figure 20.3 - Using the substitution lookup in a Devanagari string.

Specifications for creating a wide range of OpenType fonts used in complex scripts are available on the Microsoft Typography site at http://www.microsoft.com/typography/tt/tt.htm. Other tables that provide pointers to scripts supported in a font are the cmap and OS/2 tables. However, the information these tables give is not used for OpenType Layout purposes.


Issues

One issue to keep in mind is that without OpenType, achieving basic necessary typographic quality for many complex scripts is close to impossible. Unless fonts for such complex scripts are Unicode-encoded and contain layout tables, Uniscribe and other operating-system services will not recognize them. In other words, shaping engines in Uniscribe cannot be expected to properly shape scripts in fonts that are not encoded in Unicode.


International Best Practices

There are several best practices you should keep in mind when working with OpenType fonts. Additional information is available at the Web sites that are listed in this section.

Pay Attention to All Linguistic Features

OpenType fonts for complex scripts should implement all linguistic features that are identified for the script or language system that they support. A list of all registered OpenType features is available at http://www.microsoft.com/typography/otspec/feattags.htm.

Note the Order of Lookups

Lookups are applied exactly in the order that they appear in the layout tables. Thus it is of vital importance for this order to match the way the designer intended for lookups to be applied.

Consider Storage Efficiency

When more than one format can be used to construct a lookup, use the format that results in the greatest storage efficiency. Also, in cases where more than one lookup is meant to substitute different glyphs-and if the lookups share the same context-find out if their contents can be merged into one lookup.

Use Device Tables to Position Diacritics at Very Small Point Sizes

Device tables improve positioning for a short range of very small point sizes. These tables can also be used for subtle positioning adjustments at a specific point size that do not occur at other point sizes. For instance, there might be an adjustment at 51 pixels per em (ppem) that does not occur at 50 ppem. However, device tables do add to the size of the GPOS table. Therefore, if a device table is being used for positioning a glyph across a very large range of point sizes, you should examine if the anchor on that glyph needs to be moved to a better (x, y) location.

Familiarize Yourself with All Recommendations for OpenType Fonts

You should be aware of guidelines on byte ordering, outline formats, file names, as well as table alignment and length. These recommendations are available at http://www.microsoft.com/typography/otspec/recom.htm. In addition to these guidelines, the specifications for creating OpenType fonts that support complex scripts are an essential reference.


Alternatives

Apple Advanced Typography fonts, which make use of layout capabilities of the QuickDraw GX Layout model, are an alternative solution to OpenType fonts. Both GX and OpenType use the cmap table to map character code points to glyphs indices, after which all processing is done using glyph indices. There are several differences between the GX model and OpenType:

  • GX provides support for bidirectional processing and reordering (for some South Asian scripts). With OpenType, Uniscribe or the text-processing application performs these activities.
  • GX uses a property table in the font to ascertain direction and to determine any other property for each glyph in the font. OpenType infers properties based on the Unicode point to which the glyph is mapped.
  • GX uses the metamorphosis (also called the "mort") table for substitutions that involve building up ligatures from their component glyphs, using noncontextual substitutions (such as a swash variant), or making contextual changes in the form of a letter, which occurs in Arabic. OpenType lists all substitutions in the GSUB table.
  • GX uses finite state machines to identify context. OpenType uses string matching.


Additional Resources

For more information on OpenType fonts, see the following resources:

  1. http://www.microsoft.com/typography/tt/tt.htm. "Specifications." Provides a collection of all specifications to date. This includes specifications that Microsoft has made public in order to create OpenType fonts for complex scripts like those used in Arabic and Indic languages.
  2. http://www.microsoft.com/typography/otspec/default.htm. "OpenType Specification." Serves as the basis of the specifications just mentioned.
  3. http://www.microsoft.com/typography/developers/volt/default.htm. "Visual OpenType Layout Tool." Describes a tool for adding OpenType Layout tables to fonts and tells how to download it.
  4. http://www.microsoft.com/typography/default.asp. "Microsoft Typography." Contains news, updates, specifications, and information on fonts; offers free utilities and TrueType fonts.
  5. http://developer.apple.com/fonts/WhitePapers/GXvsOTLayout.html. "Comparing GX Line Layout and OpenType Layout." Provides comparative details of both technologies.


Summary

Expanding upon the capabilities of TrueType, OpenType provides better support for international character sets, protection for font data, and advanced typographic control. OpenType fonts allow a rich mapping between characters and glyphs, thus enabling support for ligatures, positional forms, alternates, and other substitutions. Another benefit of OpenType fonts is their layout tables for multilingual text, which give font designers greater typographic control. They are also portable across platforms and can support all common encoding systems.

OpenType fonts for complex scripts use Windows operating-system services such as Uniscribe to shape and position glyph strings that character strings are mapped to. Uniscribe, in turn, uses the OpenType Layout services library, which is a set of helper functions to lay out the glyph string during text processing.

To take full advantage of OpenType fonts, pay attention to the order of lookups. When you construct lookups, try to use a format that will yield the greatest storage efficiency. In general, be current on guidelines for creating Open-Type fonts-including recommendations on usage of OpenType features, byte ordering, outline formats, file names, as well as table alignment and length.


Chapter 21: Rich Edit

This chapter discusses the benefits of using Rich Edit when creating world-ready applications. Included is an overview of what this technology provides, where it is available, what international features, issues, and international best practices are associated with it, and what alternatives are available. Finally, the chapter will supply you with additional resources regarding this technology.


Overview of the Technology

Glossary
  • Text Object Model (TOM) interfaces: A substantial set of text-manipulation interfaces. Text solutions such as Microsoft Word and rich edit controls support the TOM feature set. Since rich edit controls ship with Microsoft Windows operating systems, they are the standard means of obtaining TOM functionality.

Rich Edit is a component that provides a variety of edit functionality in a single, worldwide binary that runs on Microsoft Windows 95/98/Millennium Edition (Me) and on Microsoft Windows NT/2000/XP. The edit controls can be plaintext or rich text, single or multiline, and can support Unicode encodings and standard Windows character sets (125x, Shift-JIS, GB 2312-80, Big-5, Hangul, Thai).

Rich edit controls are windows in which the user can enter, edit, format, print, and save text. The text can be assigned character and paragraph formatting, and can include embedded Component Object Model (COM) objects. Richedit controls support almost all of the messages and notifications used with multiline edit controls. Thus applications that already use edit controls can be easily changed to use rich edit controls. Additional messages and notifications enable applications to access the functionality unique to rich edit controls.

Rich edit controls have a multilevel Undo feature, message and COM interfaces, considerable compatibility with Word, rich character and paragraph properties, a zoom feature, support for Input Method Editors (IMEs), speech and handwriting input, rich complex-script support (for bidirectional languages, Indic languages, Vietnamese, and Thai), pagination, multilevel tables, autocorrection, hyphenation, kerning, and auto-URL detection. Also available within rich edit controls are East Asian features such as vertical text, font binding, and support for Unicode surrogate pairs and most Unicode 3.0 scripts. This chapter will give you a closer look at some of these features and capabilities.

Clients of Rich Edit include Microsoft Office dialog boxes, WordPad, Microsoft Outlook Rich Text Format (RTF) editor, Microsoft Pocket Word, eBooks, Microsoft FrontPage source-view functionality, and Microsoft Visual Basic edit controls. Itself a component, Rich Edit uses other components including Uniscribe (an advanced-typography line-layout component), Microsoft Windows Text Services Framework (TSF), a special input services component, and various callbacks for access to autocorrect, hyphenation, and ClearType libraries. As a result, Rich Edit scales from simple single-line, plaintext controls used in dialog boxes to quite sophisticated rich-text instances with surprisingly high quality typography.

The following demonstrates how to create a rich edit control. It also explains ways to enhance the rich edit control, such as by using certain messages and TOM interfaces.

Creating a Rich Edit Control

The sample code provided in this section offers assistance as you work with Rich Edit to create rich edit controls. Somewhere in your code you need to load the appropriate rich edit library. For Rich Edit 1, use:

LoadLibrary(TEXT("riched32.dll"));

For Rich Edit 2, 3, or 4, use:

LoadLibrary(TEXT("riched20.dll"));

For Rich Edit 4.x, use:

LoadLibrary(TEXT("msftedit.dll"));

One way to create a rich edit control is to use the following code:

HWND CreateRichEditWindow(
   HWND hwndParent,
   LPRECT prc,
   DWORD dwStyles,
   DWORD dwExStyles.
   BOOL fUnicode,
   HINSTANCE hinst)
{
   if(!dwStyle)
      dwStyle = ES_AUTOHSCROLL | ES_AUTOVSCROLL | ES_MULTILINE |
                ES_NOHIDESEL | ES_SAVESEL | ES_SELECTIONBAR |
                WS_CHILD | WS_CLIPCHILDREN | WS_HSCROLL |
                WS_VISIBLE | WS_VSCROLL;

   HWND hwnd = CreateWindowEx(
           dwExStyles,
           Unicode ? TEXT("RichEdit20W") : TEXT("RichEdit20A"),
           0,
           dwStyle,
           prc->left,
           prc->top,
           prc->right - prc->left,
           prc->bottom - prc->top,
           hwndParent,
           0,
           hinst,
           0);
   return hwnd;}

Note that instead of registering the old windows class names, known as "RichEdit20W" and "RichEdit20A," Rich Edit 4.1 (or Msftedit.dll) uses a new window class called "RichEdit50W." Thus for the previous example, you would use the "RichEdit50W" window class name in the CreateWindowEx call when working with Rich Edit 4.1.

The window styles in Rich Edit are handy but rather limited. To enhance the rich edit control, messages such as EM_SETCHARFORMAT, EM_SETPARA-FORMAT, and TOM interfaces are quite useful. You can also create a rich edit control in windowless mode by executing CreateTextServices(), which is exported by the rich edit dynamic-link library (DLL) and returns a pointer to an ITextServices object.

To use the TOM interfaces-which consist of a set of six Common Object Model dual interfaces-you can get ITextDocument and ITextRange interfaces from a rich edit window at the point pt by using the following code:

HWND hwnd;
ITextDocument *pDoc;
ITextRange *pRange;
POINT pt;
IUnknown *pUnk = NULL;
GetCursorPos(&pt);
hwnd = WindowFromPoint(pt);
SendMessage(hwnd, EM_GETOLEINTERFACE, 0, (LPARAM)&pUnk);
if(pUnk &&
   pUnk->QueryInterface(IID_ITextDocument, &pDoc) == NOERROR)
   {
      pDoc->RangeFromPoint(pt.x, pt.y, &pRange);
      //      Continue with rest of program.
   }

Indeed, Rich Edit provides versatility and scalability to its many clients. In addition, it is widely available, as you will see in the next section.


Availability

All Microsoft Windows operating systems (including Microsoft Windows CE) can run some version of Rich Edit. Rich Edit 1 is delivered in the file Riched32.dll, whereas Rich Edit 2.x, Rich Edit 3, and Rich Edit 4 are delivered in Riched20.dll. Starting with version 4.1 of Rich Edit (for Microsoft Windows XP SP1), the DLL is called "Msftedit.dll." Nevertheless, Rich Edit 4.x can run on all previous operating systems and is quite backward-compatible. Rich Edit 3 is used to emulate all the localized versions of Rich Edit 1 for Windows 2000 and Windows XP. Both Rich Edit 3 and Rich Edit 4 are single, worldwide binaries, though Rich Edit 4 has significant international improvements over Rich Edit 3 as well as TSF input support. Any localization needs (such as for pop-up menus) are delegated to the client, so that Rich Edit has almost no localized code. Table 21-1 summarizes the minimum Rich Edit support on various operating systems. All operating systems have at least Rich Edit 1.

Table 21-1 Rich Edit support on various operating systems.

Operating System | Rich Edit Version | Features
Windows 95 | 1 | Only localized versions, which are not Unicode-based.
Windows 98 | 1, 2 | 2 is Unicode-based for Western European and East Asian languages.
Windows Me | 1, 3 | 3 is Unicode-based for Western European, East Asian, and bidirectional (BiDi) languages.
Windows NT 4 | 1, 2 | 2 is Unicode-based for Western European and East Asian languages.
Windows 2000 | 3 (emulates 1) | Unicode-based for Western European, East Asian, BiDi, and some South East Asian languages.
Windows XP | 3 and later (emulates 1) | Unicode-based for Western European, East Asian, BiDi, and some South East Asian languages.


International Features

The fact that Rich Edit is so widely available, combined with its numerous international features, makes it a convenient and efficient technology as you develop and create world-ready applications. Some of the international features associated with Rich Edit, discussed in the sections that follow, include font binding, Unicode surrogate pairs, interfaces for using rich edit controls, and support for IMEs.

Starting with version 2, Rich Edit has been based on Unicode. This gives considerable international capability for Western European and East Asian languages. (For more information on Unicode, see Chapter 3, "Unicode." ) Rich Edit 3 added support for BiDi languages, Devanagari, Tamil, Thai, and Vietnamese. Rich Edit 4 added support for most of the remaining Unicode 3.0 scripts as well as for surrogate pairs and TSF. Because Rich Edit has many clients, a major principle has been to provide a single, worldwide binary that runs on all the operating systems. This is a need of international applications in general. (For more information on the advantages of creating a single, worldwide binary, see Chapter 1, "Understanding Internationalization," and Chapter 2, "Designing a World-Ready Program." )

Font Binding

A little rich-text functionality is necessary for displaying Unicode Chinese, Japanese, and Korean (CJK) plaintext unambiguously in order to account for glyph variations that exist among Simplified Chinese, Traditional Chinese, Japanese, and Korean. Rich-text functionality handles font choices and language-dependent glyph variants. For example, in plaintext dialog boxes, East Asian users usually expect to see Chinese characters displayed in only one of the four languages just listed. Heuristics often are adequate to make the right choice, but when a string contains only Chinese characters belonging to two or more of these languages on a computer equipped with these languages' respective fonts, ambiguity results. The only way to resolve this uncertainty is to use language tagging.

When Rich Edit first detects any character belonging to a complex script (such as for Arabic, Thai, Hebrew, or Indic languages), it dynamically binds to the advanced-typography line-layout component along with Uniscribe. Uni-scribe contains a number of shaping engines including one for Korean jamos. This particular engine can be used to display all Korean characters, whether modern or ancient. (For more information on Uniscribe, see Chapter 5, "Text Input, Output, and Display.") In addition, a Uniscribe shaping engine for non-spacing combining marks is very useful for mathematical text. This type of engine accesses OpenType font tables to attach combining marks at the appropriate positions on the base character.

There can be disparity between typed text and set text. When a user types in text using a keyboard charset, the edit engine knows the charset and therefore can insert accurate Unicode text-including refinements such as which CJK glyph variant to use. If the client gets the plaintext from the control (as Unicode or non-Unicode), and then sets the text again, the original rich-text clues provided by the keyboard are gone. In such contexts, it would be handy to have writing-system tags like Latin 1, Greek, Russian, Arabic, Japanese, and so on. Language tags-for example, the lower 16 bits of a Microsoft Win32 locale ID (LCID)-are usually more specific and would also work, but are superfluous unless proofing tools are to be supported.

Much of Unicode consists of characters that belong to a set of writing systems such as Latin 1, Greek, Russian, Arabic, Japanese, and so on. Rich Edit effectively associates a font bundle with each position in a document. A font bundle is a set of fonts corresponding to Unicode characters of a particular writing system. A plaintext document has a single font bundle, which can be computed on the fly from other information. As characters are inserted, each is assigned to a script in a context-dependent way. For Chinese character assignments, surrounding characters are checked for kana and hangul in an attempt to use Japanese or Korean fonts instead of Chinese. Context allows you to ascertain the script to which neutral characters (such as blanks) and digits belong. In addition, the keyboard language, especially IMEs, can provide strong binding clues. Theinserted characters are formatted with fonts assigned to a particular writing system unless the current font supports the required writing system. Whether a font can support a given writing system can usually be efficiently determined by getting the font signature. (For more information, see the documentation on the GetTextCharsetInfo() function at http://msdn.microsoft.com.) Rich Edit 4 supports approximately 50 writing systems from Unicode 3.1.

Unicode Surrogate Pairs

Rich Edit uses a pair of UTF-16 Unicode surrogates to represent a single supplementary-plane character. This approach is commonly used in Microsoft software and gives a smaller instance size than using 32-bit characters (UTF-32), but it causes complications beyond the measurement and display of characters. (For more information on surrogate pairs, see Chapter 3.) Arrow-key handlers and other methods that change the caret character position should not end up between the lead and trail surrogates. Input methods need to map to the surrogate pair. Case changes, line-breaking rules, sorting, file formats, and backing-store manipulations, in general, have to recognize and deal with surrogate pairs.

Luckily the choice of the surrogate code ranges makes surrogate pairs easy to work with relative to multibyte encoding systems, since you can easily tell if a code value is a lead surrogate, trail surrogate, or neither. Microsoft surrogate-pair support uses fonts with a new 21-bit cmap that allows mixing characters from Unicode's 17 planes. This support is consistent with TrueType's current 16-bit glyph indices so that the TrueType rasterizer doesn't need to be revised, which limits the total number of glyphs in a font to 65,536. To handle more glyphs, multiple fonts need to be used. This is a fairly straightforward process because of the font binding methods that Rich Edit uses. As Unicode adds support for new scripts, new fonts will need to be created to represent these scripts' glyphs. These additional fonts can include new tables, including a 21-bit cmap. Fortunately, Rich Edit clients can typically ignore these problems with surrogate pairs, since Rich Edit and its companion DLLs provide automatic resolution.

Interfaces

There are four main ways to use a control within Rich Edit 2.x, Rich Edit 3, or Rich Edit 4.x :

  • Messages
  • File read and write (plaintext or RTF)
  • Text Object Model methods
  • ITextServices methods

These four options are described in the following sections.

Messages

Rich Edit handles system messages including keyboard messages like WM_KEY-DOWN, WM_CHAR, mouse messages like WM_MOUSEMOVE, WM_LBUTTON-DOWN, and clipboard messages like WM_COPY, WM_CUT, and WM_PASTE. Rich Edit also supports most of the system edit messages (as defined in Winuser.h) except for EM_GETHANDLE, EM_SETHANDLE, EM_FMTLINES, and WM_GET-FONT.

Rich Edit has many of its own messages to yield access to features. For example, EM_INSERTTABLE (defined for Rich Edit 4) allows you to insert nested tables with cells that can contain multiple paragraphs. Rich Edit messages are defined in Richedit.h and have been included in the Microsoft Windows Platform SDK, available from http://msdn.microsoft.com.

File Read and Write

Rich Edit reads and writes text in a variety of plaintext and rich-text formats. Plaintext characters are assigned to scripts and bound to appropriate fonts. Rich-text files typically have appropriate fonts, but Unicode characters that aren't assigned to a particular script are also subject to font binding when Rich Edit reads text that contains such characters.

Text Object Model Methods

TOM comprises a set of six Common Object Model dual interfaces. The Common Object Model dual interface obeys rules that let it be used by a variety of clients, ranging from Microsoft Visual Basic, Microsoft C, Microsoft C++, and Java programs to simple Automation containers (formerly known as "OLE Automation containers"). Clients can run TOM from Visual Basic by creating Visual Basic scripts and from Java by creating Java scripts. This is flexible and relatively easy to do. When efficiency really counts, clients can access TOM directly from C and C++.

The top-level TOM object is defined by the ITextDocument interface, which has methods for creating and retrieving objects lower in the object hierarchy. For simple plaintext processing, you can obtain an ITextRange object from an IText-Document object and perform most editing tasks with that. If you need to manipulate rich-text formatting, youcan obtain ITextFont and ITextPara objects from an ITextRange object. ITextFont provides the programming equivalent of the Font dialog box (on the Format menu) in Word, and ITextPara provides the equivalent of the Paragraph dialog box (on the Format menu) in Word. In addition to these four lower-level objects, TOM has a selection object (ITextSelection), which is just an ITextRange object with selection highlighting and some additional methods oriented toward the user interface (UI). The range and selection objects include screen-oriented methods that enable programs to examine text on the screen or to view text that can be scrolled onto the screen. These capabilities help make text accessible to the blind, for example.

ITextServices Methods

Windowless rich edit controls provide rich-text editing support for windowless objects. Instead of creating a windowed rich edit control, you create a text services object that uses ITextServices/ITextHost interfaces to provide access to the rich edit functionality. The control then works somewhere inside the host window, and many controls can belong to the same host window. This can greatly reduce the instance size of a set of controls as well as centralize the administration of the controls.

Support for Input Method Editors

Glossary
  • Active Input Method Manager (IMM): An ActiveX control that provides limited IMM service on non-Asian language versions of Windows 95, Windows 98, Windows Me, and Windows NT 4 platforms. It is replaced by the more general Text Services Framework in Windows XP. Active IMM is also known as "Global IME."

Rich Edit has supported IMEs for years. (For more information on IMEs, see Chapter 5.) Rich Edit 4.x integrates the latest Microsoft advances in this area by giving native support to the new Text Services Framework. TSF, which is shipped with Office XP and Windows XP, enables an application to extend speech, handwriting, and East Asian input support across all localized versions of Win32 platforms. (For more information on TSF, see Chapter 23, "Microsoft Windows Text Services Framework [TSF].") Rich Edit also has a Unicode input method based on the Alt+X hot key (described later in this section) and a standard built-in (overrideable) hot keys for Cut, Paste, Copy, Undo, Redo and so on.

Internally, Rich Edit's IME capability has been factored out into an independent module that communicates with the Rich Edit engine via the TOM interfaces. It supports both Level 2 and Level 3 IMEs. Level 2 IMEs allow users to enter a candidate in a window or box before sending the final text to applications. Many of the Chinese IMEs are Level 2 IMEs. Level 3 IMEs provide applications with composition characters while the user is typing. Rich Edit's input services also support the older Active Input Method Manager (IMM). IME features include the following:

  • Reconversion. This feature enables the user to convert the final string back to composition mode, allowing easy selection of a different candidate string. In the past, the user needed to delete the final string first and then type in a new string to get to the correct candidate.
  • Document feed. This feature provides Microsoft IME 2000 and IME 2002 with the text for the desired paragraph, which allows both IMEs to have more accurate conversion during typing.
  • Mouse operation. This feature allows the user to have better control over the candidate and UI windows during typing.
  • Caret position. This feature provides the current caret and line information, which IME 2000 and IME 2002 use to position UI windows (such as a candidate list).

A handy hexadecimal-to-Unicode entry method works with WordPad 2000, Office edit boxes, rich edit controls in general, and Microsoft Word 2002. Basically, you type a character's hexadecimal code in ASCII and then type Alt+X. The hexadecimal code is replaced by the corresponding Unicode character. The Alt+X can be a toggle (as in Microsoft Office XP). That is, press Alt+X once to convert the hexadecimal code to a character, and press Alt+X again to convert the character back to a hexadecimal code. If the hexadecimal code is preceded by one or more hexadecimal digits, you need to select the code so that the preceding hexadecimal characters aren't included. The code can extend up to the value 0x10FFFF, which is the highest character in the 17 planes of Unicode.


Issues

Despite the numerous benefits and features that Rich Edit supplies, there is one issue that you should be aware of. Rich Edit's COM interfaces have extensive error messages that can help a client figure out what went wrong. The message interface is weak in this regard, and errors typically only result in ignoring an invalid message.


International Best Practices

Rich Edit 3 and earlier are documented in the Windows Platform SDK. Starting with Windows XP Service Release (SR) 1, Rich Edit 4.1 will be documented in the Windows Platform SDK. Accordingly, the following offers a glimpse at only a few of the most pertinent practices.

Consider Backward Compatibility with Non-Unicode Text

If all text were Unicode and if all display facilities supported Unicode, the Rich Edit code base would be simpler. But a Unicode text engine has to be able to import and export text in other standards, which are defined by their code pages. If you have non-Unicode plaintext, which code page should you use to convert to and from Unicode? On localized systems, the system code page is the most likely answer. However, you can enter multilingual text using keyboards in a variety of languages that are either encoded with Unicode or with multiple code pages. What code page would you use to search for such text? There's no perfect answer, but the best choice seems to be to use the code page of the keyboard you are working with at that particular moment.

Even Unicode has a number of coding schemes, such as UTF-16 big-endian, UTF-16 little-endian, UTF-8, and UTF-32. (For more information, see Chapter 3.) If text begins with a Unicode byte-order mark (BOM), use the appropriate conversion. UTF-8 can also be recognized without its BOM with pretty high accuracy, but the algorithm is fairly complicated, whereas the algorithm for handling a leading UTF-8 BOM is easy. If text begins with a rich-text header such as {\rtf, <HTML> or <!doctype HTML.>, you can route it to the appropriate conversion routine. Rich Edit employs a combination of these schemes to read non-Unicode plaintext.

Take Font Sizing into Account

In dialog boxes, 8-point Latin characters are commonly used. But 8-point Chinese characters are hard to read, so it's better to use 9 points in combination with 8-point Latin characters. Latin characters have bigger descenders than Chinese characters, since the latter only need room for an underline. Combining 8-point Latin characters with 9-point Chinese characters and keeping the same baseline increases line height beyond 9 points-to about 10 points-since an 8-point Latin descender is bigger than a 9-point Chinese descender. This can shift the text too high in a dialog box originally designed to only handle one language. Thai characters offer particular problems, since ordinarily they are displayed as14-point characters. When mixed with Western characters, Thai characters need correspondingly larger Western fonts. (For more information on font sizes and character height, see Chapter 5.)

Know How to Handle Multicode Sequences

Glossary
  • Combining-mark sequence: An alphabetic base character followed by one or more combining-mark characters such as acute and grave accents.
  • Diaeresis: Two dots placed over a vowel to indicate that the vowel is pronounced as a separate syllable (as in the word "naïve"). Typically used when two vowels are adjacent, but should be pronounced separately rather than as a diphthong.
  • Caret: The blinking line indicating the space into which you insert text.

Unicode surrogate pairs, carriage-return line-feeds (CRLFs), and nonspacing combining-mark sequences are multicode characters (more than one 16-bit quantity) that require special treatment, both for display and for cursor navigation. For basic display purposes, combining-mark sequences can be rendered approximately by standard system calls and fonts that support the combining marks. More elegant display requires an OpenType layout engine, which has tables specifying where to attach specific combining marks. For example, a diaeresis should be placed higher on an "I" than on an "i."

In terms of cursor navigation, simple caret movement across combining-mark sequences precludes you from ending up inside such a sequence. So if you type a Left Arrow key whenever the caret immediately follows such a sequence, you should find the caret immediately in front of the sequence. A Backspace key should delete one combining mark at a time. Mouse-cursor hit testing should exit the selection at the beginning or at the end of a combining-mark sequence-never within such a sequence (at least with a simple model). A more elegant model might allow selection and editing of the individual combining marks, but this sort of capability is relatively hard to implement. Rich Edit does let you use the Backspace key to delete one combining mark at a time, as you work backward from the end of the character sequence.


Alternatives

Alternatives to Rich Edit include system edit controls, combo boxes and list boxes, the Trident module (Mshtml.dll), and direct use of Graphics Device Interface (GDI) and User functions. Each alternative has advantages and disadvantages, and all are used both inside and outside of Microsoft. International text editing is very complicated in general, so writing appropriate editors is far from being a trivial task.


Additional Resources

For more information about Rich Edit, see the following resources:

  1. http://msdn.microsoft.com. Provides documentation regarding Rich Edit, including rich edit controls and interfaces, windowless rich edit controls, and various versions of Rich Edit.
  2. http://msdn.microsoft.com/library/default . Contains information on the Text Object Model.
  3. http://support.microsoft.com/default.aspx?scid=kb;en-us;Q308310 . Addresses issues with redistributing Rich Edit.


Summary

Rich Edit provides excellent support as you create world-ready applications. You can use Rich Edit to provide edit controls in a single, worldwide binary to the Windows operating systems. The edit controls can be plaintext or rich text, single or multiline, and can support Unicode and non-Unicode character set encodings. Because Rich Edit is based on Unicode, it has considerable international capability for Western European, East Asian, Middle Eastern, Cyrillic, Indic, and other languages. If you use rich edit controls based on Rich Edit 3 or later, you don't have to worry about handling complex scripts or IMEs, since this kind of support is built into Rich Edit. You also don't have to figure out which fonts to use for Unicode's many characters, since Rich Edit has built-in font binding.

Other international features associated with Rich Edit include Unicode surrogate pairs and interfaces for using rich edit controls. For instance, you can use the Text Object Model interfaces, which are supported by rich edit controls, for plaintext and rich-text manipulation. These extensions allow clients to take advantage of Unicode text handling and font binding in Rich Edit.

When working with Rich Edit, remember to consider font sizing issues. For instance, mixing 8-point Latin characters with 9-point Chinese characters in dialog boxes will result in increased line height, and can cause text to shift up higher than if the dialog box were only designed to handle one language. Moreover, Unicode surrogate pairs, carriage-return line-feeds, and nonspacing combining-mark sequences will require special treatment, both for display and for cursor navigation.


Chapter 22: Microsoft SQL Server

This chapter discusses the benefits of using SQL Server when creating world-ready applications. Included is an overview of what this technology provides, where it is available, as well as what international features, and international best practices are associated with it. Finally, the chapter will supply you with additional resources regarding this technology.


Overview of the Technology

For a database to provide central data storage-which is obviously its primary task-it should have a mechanism to store international, language-specific characters and to support different clients that are running various character encodings at the same time. Microsoft SQL Server 2000 includes powerful features to support international operations and environments. Unicode support is the foundation of the extensive multilingual features of SQL Server 2000, making it a compelling database product and applications platform.


Availability

SQL Server 2000 versions are available through several channels and are offered in English, French, German, Italian, Spanish, Japanese, Korean, Simplified Chinese, and Traditional Chinese.


International Features

Glossary
  • Collation: Refers to a set of rules that determine how data is sorted and compared.

In order to store characters from a wide variety of languages, SQL Server relies on various data types. Collations also play an important role in handling multilingual data, with their ability to sort both Unicode and non-Unicode data in a consistent manner. The following sections differentiate the data types in SQL Server 2000, explain the significance of collations, and show how to retrieve information about collations.

Data Types in SQL Server 2000

Glossary
  • SQL-92: Designed by what was formerly the National Committee for Information Technology Standards (NCITS)-now known as the "Inter-National Committee for Information Technology Standards" (INCITS)-Technical Committee H2 on Database to be a standard for relational database management systems (RDBMSs).
  • Transact-SQL: A method to communicate with and access data on SQL Server. Applications that communicate with SQL Server do so by sending Transact-SQL statements to the server, regardless of an appli-cation's user interface (UI).
  • SQL Query Analyzer: A graphical user interface for designing and testing Transact-SQL statements, batches, and scripts interactively.
  • SQL Server Enterprise Manager: Allows enterprise-wide configuration and management of SQL Server and SQL Server objects.

The data types you use to store international data depend on whether you are dealing with non-Unicode data, Unicode data, date and time values, or a combination of these categories. For non-Unicode data, you would either use char, varchar, or text data types. The data types nchar, nvarchar, and ntext are used in SQL Server for Unicode data, whereas datetime and smalldatetime are used to represent date and time values. There are several areas to consider when using SQL Server 2000 data types for storing international data.

Non-Unicode Text Types: char, varchar, text

When you deal with text data that is stored in the cha r, varcha r, or text data types, the most important limitation to consider is that only information from a single code page can be stored. The exact code page depends on the collation of the column. If there is no column-level collation, the collation of the database is used. (For more information on collations, see "Collation" later in this chapter.) To determine the code page that is used for a given column, you can use the COLLA-TIONPROPERTY function, as shown in the following examples:

SELECT COLLATIONPROPERTY('Chinese_PRC_Stroke_CI_AI_KS_WS','CodePage')
936

SELECT COLLATIONPROPERTY('Latin1_General_CI_AI','CodePage')
1252

SELECT COLLATIONPROPERTY('Hindi_CI_AI_WS','CodePage')
0

In the last example, Hindi was added to point out that many locales (such as Georgian and Hindi) do not have code pages, since they are "Unicode-only" collations, which are not appropriate for non-Unicode data types.

Any time Unicode data must be inserted into these columns, the columns will be internally converted from Unicode using the WideCharToMultiByte application programming interface (API) and the code page associated with the collation. Any time a character cannot be represented on the given code page, it will be replaced by a question mark (?); therefore, random question marks are a good indication of data that has been corrupted due to this conversion. The appearance of question marks also signals that you should have used a Unicode data type. If you use a string literal of a non-Unicode type, it will be converted using the database's default code page (derived from its collation). If you want the string literal to be interpreted by the server as Unicode, you must put "N" before the string literal.

You might encounter another problem if you try to store data when not all of the characters you wish to support are contained in the code page. One of the best examples of this is the Arabic script. It supports a wide array of languages, including Baluchi, Berber, Farsi, Kashmiri, Kazakh, Kirghiz, Pashto, Sindhi, Uighur, Urdu, and more. All of these languages have additional characters beyond those in the Arabic language, which is the basis for Microsoft Windows code page 1256. These extra characters are thus converted into question marks if they are stored in a non-Unicode column with the Arabic collation. This problem occurs because, in many cases, Windows will consider a particular code page to be a "best fit" code page. That means there is no guarantee you will be able to rely on the code page to handle all characters, but it is still the code page with the closest match to the characters you are trying to support.

Unicode Text Types: nchar, nvarchar, ntext

The SQL-92 specification defines these "N" (stands for "national") data types, but does not specifically require them to be used for Unicode; the actual definition of these data types is left to the database platform or developer. In Microsoft SQL Server 7 and SQL Server 2000, these data types are defined as being equivalent to UCS-2. Remember, however, that this equivalency is specific to Microsoft SQL Server. When you work with other database servers (such as Sybase SQL Server), "N" data types do not necessarily mean Unicode.

For the storage of complex scripts, such as those used in Hindi and Tamil, it is important to note that the data is expected to be in the proper order. Many languages such as Tamil will actually specify that certain letters be reordered when the text is rendered, thus making the logical order of text that is stored in memory different from the visual order that will be seen in a UI. (For more information, see Chapter 5, "Text Input, Output, and Display.") Data should always be stored in the proper logical order for any complex-script language, which includes all of the Indic languages, Arabic, Farsi, Hebrew, and many others. The actual rendering of such data is a separate issue.

Although the "N" columns can indeed support data of any language or combination of languages, the actual sorting of this data can only be in a single collation. (For more information on the meaning and consequences of this issue, see "Collation" later in this chapter.) None of the code-page limitations mentioned previously applies to Unicode columns.

Date and Time Types: datetime, smalldatetime

These data types have no actual international meaning; they represent a date or time value with the following definitions:

  • datetime. Date and time data in the Gregorian calendar from January 1, 1753 through December 31, 9999, to an accuracy of one three-hundredth of a second (equivalent to 3.33 milliseconds or 0.00333 seconds).
  • smalldatetime. Date and time data in the Gregorian calendar from January 1, 1900 through June 6, 2079, with accuracy to the minute. smalldatetime values with 29.998 seconds or lower are rounded down to the nearest minute; values with 29.999 seconds or higher are rounded up to the nearest minute.

SQL Server rejects data that falls outside of these ranges. The actual data is stored internally as two integers (4-byte integers for datetime, and 2-byte integers for smalldatetime) that represent the date and time in question. Because the actual value has no actual connotations in regard to locale-specific formatting, it is up to the developer to define such conversions as needed.

SQL Server 2000 supports many different locale-specific conversions that can be performed at the server instead of relying on custom solutions from developers. These date styles can be accessed through the CONVERT function, which takes a data type, an expression, and an optional style.

The following example shows how the CONVERT function is used:

SELECT CONVERT(char, GETDATE(), 100) AS [100]
Aug 16 2000 11:50AM

You can then convert the data from a string to a date value in much the same manner:

SELECT CONVERT(datetime, 'Aug 16 2000 11:50AM', 100) AS [100]

It is worth noting that when you convert such dates with Style 130 (Kuwaiti or Hijri), converting to the char data type can result in data corruption if the collation is not one of the Arabic collations that use code page 1256 for Unicode conversions. Figure 22-1 shows the results of this type of conversion.

Figure 22.1 Converting character encoding in SQL Query Analyzer.
Figure 22.1 - Converting character encoding in SQL Query Analyzer.

Note that on a U.S. client computer, attempting to use the char data type results in the Arabic characters being converted into question marks, and the nchar data type rendering Arabic characters. This particular string at the top of Figure 22-1 is still not properly formatted (as it would be on an Arabic client computer) due to limitations in the SQL grid in SQL Query Analyzer.

Collation

In addition to data types, collation is another important component in handling multilingual data, whether Unicode or non-Unicode. One of the things that many people take for granted is sorted data; after all, what could be more basic than the alphabet? However, you must remember that users of non-English languages can have different sorting expectations. For example, a Spanish user might expect the letter combination "ch" to sort as a single character after the letter "h." In general, one of the most effective ways to alienate the user of an application is to be unable to correctly handle basic sorting operations.

The sorting of data is accomplished through collations, or sort orders, and a technique known as "string normalization." This type of normalization is different from the regular concept of normalization that database developers are used to, since it is a not a design issue; string normalization refers to comparing two strings so that they can be sorted, a comparison that can be optimized through the creation of indexes.

SQL Server 2000 supports objects that have different collations being stored in a single database. Separate SQL Server 2000 collations can be specified down to the level of columns, and each column in a table can be assigned different collations.

For non-Unicode columns, collation has a second meaning that is very important. Collation specifies the code page for the data and, therefore, what characters can be represented. Data can be moved between Unicode columns seamlessly, while data moved between non-Unicode columns cannot.

In SQL Server 2000, a single, consistent model was designed to handle both Unicode and non-Unicode sorts. This model supports the languages displayed in the following list.

Table 22-1 Collation in SQL Server 2000.
Collation in SQL Server 2000
Albanian | Arabic | Chinese_PRC
Chinese_PRC_Stroke | Chinese_Taiwan_Bopomofo | Chinese_Taiwan_Stroke
Croatian | Cyrillic_General | Czech
Danish_Norwegian | Estonian | Finnish_Swedish
French | Georgian_Modern_sort | German_PhoneBook
Greek | Hebrew | Hindi
Hungarian | Hungarian_Technical | Icelandic
Japanese | Japanese_Unicode | Korean_Wansung
Korean_Wansung_Unicode | Latin1_General | Latvian
Lithuanian | Lithuanian_Classic | FYRO Macedonian
Polish | Romanian | Slovak
Slovenian | Spanish (Spain) | Thai
Traditional_Spanish | Turkish | Ukrainian
Vietnamese |  |

Each of these collations is combined with a series of suffixes that help define whether there is case, accent, kana, or width sensitivity. (Note that the default setting of kana, or width sensitivity is set to insensitive.) The exact suffixes that are possible are shown in Table 22-2. Each of the 40 languages in Table 22-1 supports the 17 suffixes in Table 22-2, for a total of 680 Windows collations.

Table 22-2 Collation suffixes.
Suffix for Collation | Meaning
_BIN | Binary sort
_CI_AI | Case-insensitive, accent-insensitive, kanatype-insensitive,width-insensitive
_CI_AI_WS | Case-insensitive, accent-insensitive, kanatype-insensitive, width-sensitive
_CI_AI_KS | Case-insensitive, accent-insensitive, kanatype-sensitive, width-insensitive
_CI_AI_KS_WS | Case-insensitive, accent-insensitive, kanatype-sensitive, width-sensitive
_CI_AS | Case-insensitive, accent-sensitive, kanatype-insensitive,width-insensitive
_CI_AS_WS | Case-insensitive, accent-sensitive, kanatype-insensitive, width-sensitive
_CI_AS_KS | Case-insensitive, accent-sensitive, kanatype-sensitive, width-insensitive
_CI_AS_KS_WS | Case-insensitive, accent-sensitive, kanatype-sensitive, width- sensitive
_CS_AI | Case-sensitive, accent-insensitive, kanatype-insensitive, width-insensitive
_CS_AI_WS | Case-sensitive, accent-insensitive, kanatype-insensitive, width-sensitive
_CS_AI_KS | Case-sensitive, accent-insensitive, kanatype-sensitive, width- insensitive
_CS_AI_KS_WS | Case-sensitive, accent-insensitive, kanatype-sensitive, width- sensitive
_CS_AS | Case-sensitive, accent-sensitive, kanatype-insensitive, width- insensitive
_CS_AS_WS | Case-sensitive, accent-sensitive, kanatype-insensitive, width- sensitive
_CS_AS_KS | Case-sensitive, accent-sensitive, kanatype-sensitive, width- insensitive
_CS_AS_KS_WS | Case-sensitive, accent-sensitive, kanatype-sensitive, width- sensitive

To ensure that code pages from earlier versions of SQL Server are properly supported, many backward-compatible, SQL-specific sort orders are also included in SQL Server 2000. These SQL-specific sort orders are listed in Table 22-3. Many of these sort orders support various parts of the suffixes listed previously, but not all of the suffixes are supported.

Table 22-3 SQL-specific sort orders.
SQL-Specific Sort Orders
SQL_1xCompat_CP850 | SQL_AltDiction_CP850 | SQL_AltDiction_CP1253
SQL_AltDiction_Pref_CP850 | SQL_Croatian_CP1250 | SQL_Czech_CP1250
SQL_Danish_Pref_CP1 | SQL_EBCDIC037_CP1 | SQL_EBCDIC273_CP1
SQL_EBCDIC277_CP1 | SQL_EBCDIC278_CP1 | SQL_EBCDIC280_CP1
SQL_EBCDIC284_CP1 | SQL_EBCDIC285_CP1 | SQL_Estonian_CP1257
SQL_Hungarian_CP1250 | SQL_Icelandic_Pref_CP1 | SQL_Latin1_General_CP1
SQL_Latin1_General_CP1251 | SQL_Latin1_General_CP1253 | SQL_Latin1_General_CP1254
SQL_Latin1_General_CP1255 | SQL_Latin1_General_CP1256 | SQL_Latin1_General_CP1257
SQL_Latin1_General_CP437 | SQL_Latin1_General_CP850 | SQL_Latin1_General_CP1250
SQL_Latin1_General_Pref_CP1 | SQL_Latin1_General_Pref _CP437 | SQL_Latin1_General_Pref _CP850
SQL_Latvian_CP1257 | SQL_Lithuanian_CP1257 | SQL_MixDiction_CP1253
SQL_Polish_CP1250 | SQL_Romanian_CP1250 | SQL_Scandinavian_CP850
SQL_Scandinavian_Pref_CP850 | SQL_Slovak_CP1250 | SQL_Slovenian_CP1250
SQL_SwedishPhone_Pref_CP1 | SQL_SwedishStd_Pref_CP1 | SQL_Ukrainian_CP1251

You can retrieve the actual information about collations by using the COLLATIONPROPERTY function. In addition to the CodePage value used previously (see "Non-Unicode Text Types: char, varchar, text " earlier in this chapter), you can also pass other information types, such as LCID, which returns the Windows locale ID (returns NULL for SQL collations). You can also specify Windows ComparisonStyle (returns NULL for both Binary and SQL collations). This information can be used to verify that there is indeed an equivalence between string normalization in Microsoft Windows 2000 and SQL Server 2000 for all of the Windows collations. All of the available collations can be returned by using the fn_helpcollations() function, as shown here:

SELECT * FROM ::fn_helpcollations()

This query returns 753 rows in SQL Server 2000. Additional collations cannot be added, unless they are added in future versions of SQL Server.

In SQL Server 2000, collations can be specified at multiple levels, including the server level, the database level, the column level, and in an expression.

At the Server Level

When installing an instance of SQL Server, the default collation of that instance can be configured during setup. The default collation of the instance also becomes the default collation of the system databases: master, model, tempdb, msdb,and distribution.

After a collation has been assigned to any object other than a column or database, you cannot change the collation except by dropping and re-creating the object. This can be a complex operation. To change the default collation for an instance of SQL Server 2000 you must:

  1. Make sure you have all of the information or scripts needed to re-create your user databases and all of the objects in them.
  2. Export all of your data using a tool such as a bulk copy.
  3. Drop all of the user databases.
  4. Rebuild the master database specifying the new collation.
  5. Create all of the databases and the objects in them.
  6. Import all of your data.
Note
Instead of changing the default collation of an instance of SQL Server 2000, you can specify a default collation for each new database you create.

To query the server collation, use the Transact-SQL SERVERPROPERTY function as shown here:

SELECT CONVERT (char, SERVERPROPERTY('collation'))

At the Database Level

When a database is created, the COLLATE clause of the CREATE DATABASE statement can be used to specify the default collation of the database. If no collation is specified during database creation, the database is assigned the default collation of the model database. The default collation of the model database is the same as the default collation of the instance of SQL Server.

The collation of the database can be changed by using the ALTER DATABASE statement. For example:

ALTER DATABASE myDB COLLATE Greek_CS_AI

The current collation of the database can be retrieved by utilizing the following statement:

SELECT CONVERT (char, DATABASEPROPERTY('database_name',
'collation'))

At the Column Level

When creating a table, collations for each character-string column can be specified by using the COLLATE clause of the CREATE TABLE statement. If no collation is explicitly specified, the column is assigned the default collation of the database.

Column collation can be changed by the ALTER TABLE statement. For example:

Page: 1
ALTER TABLE myTable ALTER COLUMN mycol NVARCHAR(10) COLLATE
Greek_CS_AI

In an Expression

There can be many instances when data needs to be displayed to people in different countries or regions with the required locale-appropriate sorting. SQL Server 2000 can specify collations in expressions, which allows sorting in a particular manner so that the ORDER BY clause can be language-specific. This is done by the COLLATE clause. The syntax for using the COLLATE keyword is:

COLLATE [<Windows_Collation_name> | <SQL_Collation_Name>].

For example:

SELECT name FROM customer ORDER BY name COLLATE Latin1_General_CS_AI


International Best Practices

Among the guidelines that you should follow are to always use Unicode data types-if you use multilingual data-and to standardize the site collation where possible. You should also try to deploy a Unicode server and client if you can, consider storage space and performance, and delimit identifiers for objects in system tables.

Always Use Unicode Data Types

If you must store character data that reflects multiple languages, you should always use Unicode nchar, nvarchar, and ntext data types instead of the non-Unicode char, varchar, and text data types. There are significant limitations associated with char, varchar, and text data types, since only information from a single code page can be stored.

The other benefit of using Unicode data types is performance gain, which is due to fewer code-page conversions occurring. Here are some representative scenarios:

  • If you are running Windows 2000 or Windows XP, the kernel is expecting Unicode data; thus non-Unicode columns will have to be converted in many cases, such as when you display data or use the operating-system services.
  • If you access data via Microsoft OLE DB or Microsoft Open Database Connectivity (ODBC) version 3.7 and later, these data access methods are expecting Unicode and will convert to and from Unicode and non-Unicode during transition. This conversion will consume additional resources and time.
Note
To realistically evaluate performance issues, you must test to get conclusive data about the situation.

Standardize the Site Collation Where Possible

If you always choose collations with the same case and accent sensitivity, the user will experience consistent behavior across all systems. Most SQL Server 2000 sites choose to be case-insensitive and accent-sensitive. Case sensitivity also applies to the names of SQL Server objects; therefore, if you specify case- sensitive collations, all users must specify the correct case when querying the database.

Always keep in mind that standardizing the site collation helps eliminate the need for explicitly specifying a collation in a character or Unicode expression, because the default collation in an expression can vary depending on the current database setting for the connection. This means that the result of a Transact-SQL statement can have different collations when executed in the context of different databases. If you inevitably have to work with objects that have different collations and code pages, you must code your queries to comply with the rules of collation coercion.

Deploy a Unicode Server and Client If Possible

In a very small number of cases, everything you do with a SQL Server will happen on the same computer on which the server resides, and only SQL Server tools such as SQL Query Analyzer or SQL Server Enterprise Manager will be used. In most cases, however, the server will be interacting with other servers or clients, and might be using one or more data-access standards. SQL Server has basically two types of clients:

  • Unicode clients-OLE DB, and ODBC versions 3.7 and later
  • Non-Unicode clients-ODBC version 3.6 and earlier, and DB-Library

The possible scenarios of server and client connections, along with certain areas you will need to consider, are presented in the sections that follow.

Unicode Server and Client

This is the ideal type of configuration. By keeping data in Unicode throughout the process, you can guarantee the best performance and protection from corruption of retrieved data. This is the case with Microsoft ActiveX Data Objects (ADO) and OLE DB.

Unicode Server and One or More Non-Unicode Clients

In this type of configuration, you might not have any problems storing data, but there is obviously a serious limitation when it comes to bringing the data to the client and using it. The client code page must be used to convert the Unicode data at some point.

Non-Unicode Server and Unicode Client

This is not an ideal configuration for multilingual data, because you will not be able to keep such data on the server. At the very least, however, you can be sure the data will show up properly. This configuration has all the limitations of the previous case, without the risk of corrupting received data due to invalid conversions. An example is when a SQL Server 2000 database defines a linked server to a Microsoft SQL Server 6.5 database. All information that is received from the server running SQL Server 6.5 will be valid, but do not try to insert any data that is outside of the code page!

Non-Unicode Server and Client

This is the most limiting configuration, because you are basically restricted to a single code page at all times.

Consider Storage Space and Performance

Ideally, every column is defined with one of the Unicode data types; however, doing so when you do not need to support multilingual data can create issues related to storage space and speed.

Storage Space Issues

The actual amount of space required for Unicode data types is 2 bytes per character; the amount of space for non-Unicode data types is 1 byte for all non-DBCS text and 2 bytes for Asian languages that use DBCS. Therefore, unless your data is on one of the Asian code pages, you will be using twice as much space to store the data. This must be considered when you upgrade existing databases or when you are deciding on the proper data types of new projects. If you are storing data only in a column that is on a single (non-Asian) code page, you might prefer not to use Unicode so you can save disk space and memory.

Speed Issues

In addition to storage space, you will also need to consider how performance is affected when you use Unicode data types unnecessarily. The speed issue is a complicated one. Here are some of the issues:

  • Indexes on columns within tables or views cannot be made collation-specific. If you use the COLLATE keyword in an "ORDER BY" expression, the SQL Server query optimizer cannot use a colllation-specific index to execute this query, thus affecting the server's performance.
  • Additional time is needed to load the larger amount of data when you are dealing with DBCS data.
  • If you are dealing with a Microsoft Windows 95 or Microsoft Windows 98 client or server, much of the information might need to be converted from Unicode when operating-system services, such as data display, are needed.
  • If you are working between servers, database server products, or other products, the number of conversions can also play a great role in the performance battle.
  • If you are dealing with Asian languages, Unicode will actually be faster than using the language-specific DBCS code page. This is because DBCS data does not have a fixed width; it is a mixture of double-byte and single-byte characters.
  • If you are dealing with non-Asian languages, sorting Unicode data can be slower than non-Unicode data. This can be considered one cost of being able to represent global data.
Important
To realistically evaluate a performance issue, you must test to get conclusive data about the situation.

Delimit Identifiers for Objects in System Tables

The system tables in SQL Server 2000 store all the data they contain as Unicode. This minimizes the problems that can occur with different collations between databases and columns. There is no other way to deal with the fact that different databases on the same server can have any mix of Unicode and non-Unicode column names. Even if you are supporting only a single language now, SQL Server must be ready to support any language you might choose to support in the future.

When you are converting databases and servers from SQL Server 6.5 or earlier, it is easy to become concerned about the metadata that is being converted; however, there is no need to worry. The conversion to Unicode is straightforward because these earlier versions of SQL Server are only in one code page or collation, each of which is defined at the server level.

One important issue involves using identifiers for objects in the system tables. SQL Server 2000 uses the Unicode 2.0 character property definition to create the list of valid characters in identifiers. (Unicode 3.0 was not released when SQL Server 2000 development was completed.) To avoid running into issues with international characters that are not defined in the Unicode 2.0 character property definition, you should delimit your identifiers with brackets ([]) or double quotation marks ("). This prevents the server from checking for valid characters.


Additional Resources

For more information about Microsoft SQL Server, see the following resources:

  1. http://www.microsoft.com/sql. Supplies a wide range of information regarding SQL Server 2000, with links to the SQL Server magazine, newsgroups, and timely news items.
  2. http://msdn.microsoft.com/library. "Collations." September 2001. Explains how character data is stored and also discusses Unicode data and sort order.


Summary

Microsoft SQL Server 2000 is a powerful technology for supporting international operations and environments. At the heart of its multilingual features is the Unicode support available in SQL Server 2000. Through data types, SQL Server can store international, language-specific characters and support different clients that are running various character encodings at the same time.

Non-Unicode text is stored in the char, varchar,and text data types.Unicode data is stored in the nchar, nvarchar, and ntext data types in SQL Server 7 and SQL 2000. The datetime and smalldatetime data types represent various date and time values.

In addition to data types, collation is another important component in sorting and comparing multilingual data, whether Unicode or non-Unicode. In SQL Server 2000, a single, consistent model was designed to handle both Unicode and non-Unicode sorts. There are 40 languages that support 17 collation suffixes, producing a total of 680 Windows collations.

Among the guidelines that you should follow with SQL Server are to always use Unicode data types and to standardize the site collation where possible. You should also try to deploy a Unicode server and client if you can, consider storage space and performance, and delimit identifiers for objects in system tables.


Chapter 23: Microsoft Windows Text Services Framework (TSF)

This chapter discusses the benefits of using TSF when creating world-ready applications. Included is an overview of what this technology provides, where it is available, as well as what international features are associated with it. Finally, the chapter will supply you with additional resources regarding this technology.


Overview of the Technology

Text Services Framework was designed to provide applications with a next- generation input solution that supports a wide range of advanced text services including accessibility, speech, handwriting, and keyboard input, as well as proofing and reference services. These services are supported through a single application programming interface (API), which is device- and language-independent.

Through the TSF APIs, applications expose document content and meta-content to services for inspection and editing. Text services can, in turn, associate and persist state with that content. The rich exchange of information between the text services and the application document enables recognition services to provide more accurate conversion results and seamless, natural input and correction behavior. Since the text services themselves are solely responsible for implementing input behavior, they are able to present a consistent user interface (UI) across applications.

TSF Capabilities

Again, one of the general benefits of TSF is its ability to provide text services within the context of a document, which improves conversion accuracy. Data persistence, the negotiation of text services, and interfaces for creating a custom UI are other capabilities that TSF offers.

Empowering Services and Applications

TSF has access to the document buffer and can use this context to provide optimal conversion results. For example, in the scenario represented in Figure 23-1, a speech service can examine the current selection and, in this case, can see the insertion point is before the word "apple." Consequently, the speech service inserts "red" instead of "read." Additionally, TSF enables applications to share both metadata and hints related to a document object or specific range of text-such as "This is a date or number field." Metadata such as this allows recognizers to significantly narrow the recognition domain and to provide optimal accuracy.

Figure 23.1 The speech service within TSF.
Figure 23.1 - The speech service within TSF.

Data Persistence

In addition to providing context to services and applications, TSF offers further capabilities through data persistence. With data persistence, a text service can store and retrieve important state information (such as a language ID or conversion alternatives) with text in the document. For example, a user reviewing a document into which text was dictated might see "wreck a nice beach," when the actual dictation was "recognize speech." Since the audio and alternatives were stored with the document, the user can listen to the original audio to understand what was intended. Similarly, a keyboard service can store the keystrokes or perhaps phonetic data to capture the original user input.

Data persistence can also be used to simply defer recognition. For example, a handwriting service, via the TSF text interfaces, can insert the raw ink objects into the document until it has sufficient information and confidence to perform the conversion. Another possibility is to enable the user to insert handwritten input into the document (like an "analog" notepad today) until the user wants it converted.

Moreover, data persistence makes it possible to have dynamic text and seamless, natural input. Services make changes and store state directly with the application text buffer, so input compositions need never be "final." The service can always retrieve the original device and conversion information and resume its recognition. The information and data that are persisted are, by default, completely transparent to the application, though the application can choose to be picky about what it serializes.

Negotiation of Text Services and Creation of Custom UI

TSF also provides support for text services to be utilized simultaneously and for them to interact behind the scenes at the UI level. For instance, a user can dictate text while typing with a keyboard text service, and correct the dictation with a handwriting text service. In addition, TSF manages input so it does not collide and allows services to be enabled or disabled as required.

In addition to managing activation and document access, TSF provides optional interfaces that text services can use to host their UI elements (for example, mode buttons). This customized UI is incorporated into a central UI element-the system Language bar. The Language bar allows text services to expose a custom UI within it, so there is less need for individual toolbars that are specific to a particular text service. Nonetheless, text services can still choose to provide their own individual toolbars.

Architecture and Object Model

You have seen some of the benefits that TSF offers. This section examines the architecture of TSF and demonstrates how TSF works. TSF is divided into three distinct components:

  • Applications. An application is responsible for display, direct editing (such as of formatting or layout, which are functions that do not pass through the TSF layer), and storage of the text. The application provides access to its text by implementing support for TSF's ITextStore interfaces (among others), and communicates with TSF using the interfaces that the TSF manager exposes.
  • Text services. A text service, which acts as a service provider to an application, can obtain text from and write text into the application. It can also associate data and properties with a block of text. A text service is implemented as a Component Object Model (COM) in-process server that registers itself with TSF. Once registered, the user interacts with the text service using the Language bar or keyboard shortcuts.
  • TSF manager. The TSF manager acts as a communication mediator between an application and one or more text services. A text service does not need to interact directly with an application and, likewise, an application does not need to interact directly with a text service. The TSF manager is implemented by the operating system and cannot be replaced.

Figure 23-2 shows the architectural divisions in TSF, along with some of the important objects within the divisions.

Figure 23.2 Architecture of TSF.
Figure 23.2 - Architecture of TSF.

TSF is a single-process global COM object. A hierarchy of apartment-threaded objects are exposed through the main interface. At the top of the hierarchy is a thread manager (TM), inside of which are document managers (DM). The TM creates a DM at the application's request, for each stream of editable text.

ITextStore and Editing

The ITextStore interface is a means for applications to share text streams with TSF text services or Microsoft Active Accessibility clients. Each document is exposed as a stream of Unicode characters, some of which might reference embedded content. Clients can read or potentially modify the document content.

Applications are responsible for implementing either ITextStoreACP or ITextStoreAnchor. (If an object exposes both, TSF will use ITextStoreAnchor.) Clients advise their servers with either ITextStoreACPSink or ITextStoreAnchorSink. If an application decides to implement ITextStoreAnchor, it must also implement IAnchor.

Table 23-1 shows the ITextStore family of interfaces, though this is not meant to be a complete list.

Table 23-1 ITextStore family of interfaces.
ITextStore::GetSelection | ITextStoreSink::OnLockGranted
ITextStore::SetSelection | ITextStoreSink::OnStatusChange
ITextStore::GetText | ITextStore::AdviseSink
ITextStore::SetText | ITextStore::UnadviseSink
ITextStore::GetEmbedded | ITextStore::RequestLock
ITextStore::InsertEmbedded | ITextStore::GetStatus
ITextStoreACP::GetEndACP | ITextStore::QueryInsert
ITextStore::GetACPFromPoint | ITextStore::GetFormattedText
ITextStore::GetTextExt | ITextStore::GetActiveView
ITextStoreSink::OnTextChange | ITextStore::GetScreenExt
ITextStoreSink::OnSelectionChange | ITextStore::GetWnd
ITextStoreSink::OnLayoutChange |

How an application offers its ITextStore implementation differs among TSF and Active Accessibility. The details are covered in the Microsoft Windows Platform software developer kit (SDK) documentation, available at http://msdn.microsoft.com.

ITextStore generates notifications that inform the client as to when text is changed, inserted, or deleted, as well as when the selection changes, when the insertion point moves, or when the layout changes. Notifications are also used to grant a lock to the client. (Clients must have a lock to perform certain operations on a document.)

TSF documents are linear streams of (mostly) Unicode text. In addition to plaintext, documents can contain embedded OLE objects and regions. Objects in the stream are represented with the Unicode object replacement character 0xfffc. Methods are available to insert or query objects at arbitrary locations. Text services are able to hit test for pixel-to-text transformations and for text-to-pixel transformations, or they can retrieve the boundaries of the display region. TSF allows some applications to embed private data within their text streams, and transparently hides the private data from the text services.

Document Access

As stated previously, in order to perform certain operations on a document, a client must first obtain an appropriate lock, which can be read-only or read/write. When holding a read-only lock, operations that modify the document's content are not permitted. A client can request that a lock be granted immediately (synchronous) or at some later stage (asynchronous). If a client requests an asynchronous lock, it can either be granted immediately or later, at the discretion of the application.

Properties and Persistence

Properties are markup information for text. This information includes the metadata that provides hints about how the text should be displayed, alternative conversions, or the raw device from which text data was originally derived. Properties can be associated with individual characters. For example, a text service can tag runs of text as belonging to a specific language (perhaps to clear up ambiguity introduced by Unicode's Han unification). Or the text service might tag incomplete compositions (such as a single letter that will be transformed into kana as soon as another keystroke arrives, generally underlined or highlighted by an application).

Properties can also be associated with ranges of text. (See Figure 23-3.) This might include candidate lists or reading strings that other text services can examine for hints during an ongoing composition.

Figure 23.3 Associating properties with text.
Figure 23.3 - Associating properties with text.

TSF defines a set of standard properties, but services and applications are free to post arbitrary markup data. Properties are inherently public; if a service wants to associate private data with document text, it can do so with range objects or contexts.

Using TSF in Your Application

Equipped with a better understanding of TSF, how can you implement this knowledge and leverage the benefits that TSF offers? The simplest way to utilize TSF in your application is to host one of the system-provided controls that support it. (For a list of controls that support TSF and for information on how to use them in your application, see the Windows Platform SDK.)If your application uses its own editing control, you'll need to implement support for TSF. (For information on TSF APIs and for samples, see the Windows Platform SDK.)


Availability

TSF shipped with Microsoft Windows XP. A scaled-down version is also available for redistribution on Microsoft Windows 98, Microsoft Windows Millennium Edition (Me), Microsoft Windows NT 4, and Microsoft Windows 2000. Microsoft Windows 95 is not supported.


International Features

International features associated with TSF include its capacity for keeping current with new input methods that are currently gaining prominence in the marketplace. TSF also provides an efficient means of input for those who work with East Asian languages. Additionally, TSF eliminates the need for device-specific or language-specific APIs, which is a significant advantage as you enable your application to handle multilingual text input and natural-language technologies.

Support for New Methods of Input for the General User

For desktop and laptop scenarios, as well as for most languages and scripts, a standard keyboard and mouse have been relatively efficient means of interacting with the PC. As hardware has advanced in power and dimension, however, new keyboard-less and mobile devices have emerged that require new methods of input, such as speech and handwriting recognition. (For more information, see Chapter 5, "Text Input, Output, and Display.") The original Microsoft Win32 API, designed almost exclusively to take text input from the keyboard, is inadequate for these input technologies.

Speech and Handwriting Support for East Asian Languages

For typists working in languages that do not use Western scripts such as Chinese, Japanese, or Korean, a special keyboard processor known as an "Input Method Editor" (IME) is required. These IMEs are often powerful programs that utilize huge databases of information with complex algorithms that try to guess the result the user intends. (For more information, see Chapter 5.) Despite years of research, however, most IMEs today are not very easy to use. The IMEs typically require that the user type a few keys to limit the subset of candidates and then to choose from a list of those candidates before sending the result to the application. The process forces the user to switch back and forth between performing the main task (mail composition, note taking, document creation) and managing input via the UI of the IME.

Furthermore, the modal, one-directional nature of the input APIs of the past has often meant that a single typo could require the user to retype an entire phrase (sometimes with a penalty of 20 or more keystrokes). Thus East Asian users have long labored with average input rates of 13 to 18 words per minute-less than half that of their Western European counterparts. In addition to a better system for keyboard input, for these users especially, handwriting and speech have the potential to improve input efficiency significantly.

No Need for Device- or Language-Specific APIs

Enabling applications for multilingual text input and natural-language technologies on Windows has in the past meant implementing multiple, complicated, device- and language-specific APIs. Each of these APIs required a significant investment of time and energy to understand the technology, to implement support for the API, and to perform debugging. Some of these APIs also required different implementations from one language to another (such as English versus Japanese versus Korean), and many were device-dependent. Quite a few of these technologies were not completely compatible either, leaving the developer to sort out the coordination of the services and their associated UI.

TSF is designed to provide a general system framework for natural-language and input technologies; this framework factors device- and technology-specific code and behavior out to service providers. With TSF, an application implements a single ITextStore interface for access to a wide range of pluggable text services. For the most part, TSF applications do not need to be aware of or have code written for specific text services. Nor do applications need to be written for specific languages, since TSF is itself fully globalized. Once again, all text input via TSF is Unicode, and it is expected that text services will tag text with input-language IDs.


Additional Resources

For more information about Text Services Framework, see:

  1. http://msdn.microsoft.com .Contains the Windows Platform SDK, which provides excellent reference material on TSF.
  2. http://www.microsoft.com/msdownload/platformsdk/sdkupdate/psdkredist.htm . Contains the latest redistributable components for downloading.


Summary

TSF supports a wide range of advanced text services including accessibility, speech, handwriting, keyboard input as well as proofing and reference services. These services are supported through a single API, which is device- and language-independent. TSF can be divided into three components: applications, text services, and the TSF manager, which acts as a communication mediator between an application and one or more text services.

Text services supply applications with context, which can significantly improve conversion accuracy. TSF also enables the text services to associate and store state and data with the document, which facilitates dynamic content, data sharing, seamless input behavior, deferred recognition, and intelligent correction. Moreover, TSF provides support for text services to be utilized simultaneously and enables the creation of a custom UI through the optional interfaces that it supplies.

TSF offers support for speech and handwriting recognition technologies, which the traditional Win32 API cannot readily address. TSF also bypasses the relative difficulty of using IMEs for East Asian languages as well as the modal, one-directional nature of the system input API. As a result, input rates for those working with East Asian languages are notably increased. Finally, TSF applications do not need to be aware of or write code for specific text services. Nor do applications need to be written for specific languages, since TSF is itself fully globalized.


Chapter 24: Uniscribe

This chapter discusses the benefits of using Uniscribe when creating world-ready applications. Included is an overview of what this technology provides, where it is available, as well as what international features are associated with it. Finally, the chapter will supply you with additional resources regarding this technology.


Overview of the Technology

The Unicode Script Processor (USP10.dll), or Uniscribe, is the system engine used to shape and lay out complex scripts. (For more information, see Chapter 5, "Text Input, Output, and Display.") Extending beyond complex scripts, however, Uniscribe provides for Unicode text display in all Microsoft-supported scripts and languages. Uniscribe also supplies a script-independent interface for writing applications that display or edit formatted text. In addition, Uniscribe offers simple font fallback for all languages and enables support for OpenType typographic features. (For more information, see Chapter 5 and Chapter 20, "OpenType Fonts.")

In contrast, Uniscribe does not provide support for sorting, code-page translation, nor does it supply keyboards, spelling checkers, or grammar checkers. Furthermore, it does not furnish a markup language or object model for creating or manipulating formatted text.

The following sections contain information on Unicode support for text lines and text runs. You will see when to use this support, and when you should forego it, as well as what scripts and features have been added to various Uniscribe versions.

Text Line Support

Uniscribe's text line support was developed to help handle international text for the Microsoft Win32 application programming interfaces (APIs) ExtTextOut and DrawText. This support includes the ScriptStringAnalyse,ScriptStringOut, and ScriptStringFree APIs and all the ScriptStringGet functions.

Don't use Uniscribe text line support if DrawText or ExtTextOut provides sufficient functionality. These functions are easier to use, and can be faster in simple script cases. Moreover, don't use text line support for text that has formatting changes within the line. Instead use Rich Edit or the Uniscribe text run APIs mentioned in the next section.

On the other hand, you should use Uniscribe's text line support API if you need the following extra features:

  • Caret placement and hit-testing functions for authors of unformatted text edit controls.
  • Simple unformatted text justification.
  • Support for displaying (but not keying, sorting, or 8-bit coding) complex scripts and East Asian scripts on platforms that are not enabled. Note that East Asian support in Uniscribe is limited to horizontal text.

Text Run Support

Uniscribe support for text runs was developed to effectively handle international text in formatted text display (such as in Microsoft Internet Explorer) and in editing applications (such as in Rich Edit). It is also used by the text line implementation. Text run APIs include ScriptItemize,ScriptShape, ScriptPlace, and ScriptTextOut.

Don't use text run support if you can utilize an easier formatted text interface such as one that Rich Edit furnishes. Instead, use it only if you are developing an edit control or a forms display package that requires formatted text.

Uniscribe Versions

Certain scripts and features have been added to Uniscribe versions, as shown in Table 24-1. As you look at the table, note that Arabic (1) refers to Arabic support for all Arabic font types supported by the Arabic version of Microsoft Windows 95/98/Millennium Edition (Me), and Microsoft Windows NT. Arabic (2) adds support for new OpenType features required for fonts such as those with a descending baseline.

Table 24-1 Scripts and features added to Uniscribe versions.
Scripts and Features | Version | Shipped With
Arabic (1), Hebrew, Thai | 173 | Microsoft Internet Explorer 4 complex-script version
Arabic (2), Devanagari, Tamil, Vietnamese, East Asian supplementary planes | 325 | Microsoft Windows 2000, Microsoft Office 2000, Microsoft Internet Explorer 5
Gurmukhi, Gujarati, Kannada, Telugu, Old Hangul, Syriac, Thanna, East Asian vertical | 407 | Microsoft Windows XP, Microsoft Office XP, Microsoft Internet Explorer 6


Availability

Uniscribe is shipped with Windows 2000 and Windows XP, with Internet Explorer 4 and later, and with Office 2000 and Office XP. It runs on Windows 95/98/Me and on Microsoft NT/2000/XP.


International Features

Among the international features that Uniscribe provides are its support for typographic issues such as kerning, ligatures, combining marks and character reordering. As you will see in the following sections, it also handles right-to-left (RTL) and left-to-right (LTR) reading order, the many-to-many relationship that exists between characters and glyphs, and finally, caret placement.

Kerning

One typographic issue that Uniscribe handles is kerning. Every glyph in a font is designed with an advance width. This advance width should allow glyphs that are placed adjacent to each other to look as though they are uniform in density, and each glyph should appear distinct. The designer must consider every possible glyph pair and adjust each glyph's advance width-in a process called "kerning"-to achieve the best effect in the most common cases, and acceptable results in the most difficult cases.

Difficult cases occur when the shape of the adjacent sides of a pair of glyphs makes the glyphs appear either too close together or too far apart. An example of problematic spacing is the letter pair "AV." The normal spacing for this pair is too wide because the slopes of the "A" and "V" are identical, leaving too much white. (See Table 24-2.)

Table 24-2 Examples of standard spacing and kerning.
Standard spacing
With kern feature applied

The advance-width adjustment is provided by the font designer in the font's Glyph Positioning (GPOS) OpenType table. When you use Uniscribe to display text, kerning specified by a font's GPOS table is applied automatically.

Ligatures

In addition to kerning, a second typographic issue that Uniscribe addresses is ligatures, which are used in most scripts. Combination glyphs such as " [image] " are referred to as "ligatures," reflecting the fact that they are composed from separate glyphs that are tied or bound together. In English, the most common purpose of ligatures is to improve legibility. For instance, character combinations like "fi" can look amiss, because the dot on the "i" meshes with the overhanging top of the "f." Therefore, most English fonts supply a separate glyph for the " [image] " combination. Compare the word "fit" in Table 24-3 with and without the "fi" ligature glyph.

Table 24-3 Comparison of the "fi" character combination, with and without the " [image] " ligature glyph.
Using separate "f" and "i" glyphs | Using the " [image] " ligature glyph

Unicode includes separate code points for a few ligatures simply because they were already coded in pre-existing character sets. Unicode does not, and will not, encode new ligature combinations as separate code points. Applications should not use the ligature code points since they break both search and sorting algorithms. Instead, "ligaturization" is a font feature provided in the Glyph Substitution (GSUB) tables of OpenType fonts. (For more information on GSUB tables, see Chapter 20.)

Fonts can also contain optional ligatures. For example, the Palatino font includes ligatures for "ft" and "st." (See Table 24-4.)

Table 24-4 Ligatures for "ft" and "st" in the Palatino font.

Such optional ligatures can be enabled by passing OpenType feature requests to the ScriptShape and ScriptPlace APIs. Many scripts use ligatures far more extensively than English does. Wherever a ligature is a standard part of the script, Uniscribe will apply it automatically. For example, the letters "lam" ( [image]), "meem" ( [image]), and "hah" ( [image]) are displayed with a ligature in some Arabic fonts. (See Table 24-5.)

Table 24-5 Example of a ligature in Simplified Arabic and Traditional Arabic fonts.
Arabic Font | Ligature
Simplified Arabic |
Traditional Arabic |
Note
Uniscribe will apply ligature and kerning features only where they are supplied in the fonts. Since this functionality has previously been available in only a handful of applications, few fonts include GPOS and GSUB tables at the time of this writing.

Combining Marks and Character Reordering

Another aspect of multilingual text involves combining marks, since they appear in most scripts. Once again, Uniscribe provides support for dealing with this typographic feature. Combining marks range from the accent marks utilized in Western Europe to the far more complex combinations found in Indic scripts, for example. While Unicode encodes many precomposed characters, it does not do so as a general rule. The number of combinations used internationally is far more than will fit conveniently in a 16- or 20-bit character set. Unicode encodes only those precomposed forms that were already encoded in previous character sets. In Table 24-6 involving Thai, notice how the base character can take one or two combining marks, and that the position of the combining mark used in the first example has changed in the second example.

Table 24-6 Thai base character and combining marks.

In some scripts, combining marks can take up space beside the base character-to the left, to the right, or both. (See Table 24-7.)

Table 24-7 Positioning of combining marks in relation to the base character.
Combining Character | Base Character | Description
 |  | Note that the combining vowel changes shape slightly, so that its hook lands on top of the stem of the base consonant.
 |  | Note that this combining vowel displays to the left of the consonant even though it is pronounced after the consonant, just as in the first example.
 |  | Note that this leading consonant is represented by the extra hook attached at the top of the consonant.

Table 24-7 also illustrates reordering and combining-the initial reph ( [image]) is joined to the syllable without an intervening vowel by the halant/virama ( [image]). In this case, the representation for the reph is the combining hook at the top. Character reordering is script-specific and follows rules defined by Unicode.

RTL and LTR Reading Order

All Arabic and Hebrew scripts are written from right to left. However, numbers and certain other scripts are written from left to right even when part of Arabic or Hebrew text. When the reader encounters a direction boundary, the eye must skip to the far end of the reversed item (word or phrase) and read in the opposite direction.

Books that have a mixture of languages can be published with an overall LTR reading order (if the content is not predominantly Arabic or Hebrew), or with a RTL reading order (if the content is predominantly Arabic or Hebrew). The Unicode bidirectional algorithm provides a standard way to handle character ordering in the absence of higher level semantics. For example, the algorithm defines that a space inside a run of one direction has the same direction as the run; a space between runs of opposite direction takes the overall direction. (See Table 24-8.)

Table 24-8 Examples of LTR and RTL overall direction.
LTR Overall Direction |  | means good morning.
RTL Overall Direction | .means good morning |

In looking at Table 24-8, there are several things you should notice. First, the final period is displayed at the logical end of the line. Second, the content of the LTR run and of the RTL run is unchanged. Third, the LTR run and the RTL run swap places.

Many-to-Many Relationship Between Characters and Glyphs

A corollary of supporting ligatures, combining marks, and character reordering is that there is a many-to-many relationship between characters and glyphs. The term "cluster" describes the shortest run of characters wholly represented by one or more glyphs. (For more information, see Chapter 5.) For example, the code point U+0061 ("a") followed by U+00E9 ("é") might be displayed as an " [image] " ligature glyph and an acute accent glyph. In this case, the pair of code points would be considered as one cluster. Uniscribe can return a mapping table that points (lists the offset into the glyph array) to the first glyph that represents the cluster to which this code point belongs.

Caret Placement

A final typographic issue that Unicode handles is caret placement. Where should the editing caret appear when there are ligatures, combining marks, and character reordering? In some languages, the convention is that the caret is never placed inside a cluster. In other languages the caret is positioned proportionally through the cluster. Uniscribe provides APIs to retrieve the caret position associated with the leading and trailing edges of any character, and to identify the character edge nearest to any pixel offset. These APIs automatically take the script rules into account.

For scripts such as Thai and Devanagari, the caret is never placed inside a cluster. Uniscribe will snap pixel offsets to the nearest cluster boundary. For scripts such as Arabic and Hebrew, the caret advances through the cluster in proportion to the number of code points covered. Uniscribe will interpolate pixel offsets in a linear fashion.

In order to advance the caret to the correct position when the user presses the Arrow keys, Uniscribe provides the ScriptStringGetLogAttr and ScriptBreak APIs. These APIs return an array of flag bitsets, one per code point. One of the flags identifies code points in front of which the caret can validly sit.


Additional Resources

For more information on Uniscribe, see the following:

  1. http://msdn.microsoft.com. Contains an overview and details of the Uniscribe APIs. Also provides the Microsoft Windows Platform SDK, which has many helpful comments in USP10.h, the header file for Uniscribe.
  2. http://www.microsoft.com/globaldev/articles/multilang.asp. "Multilanguage text support in Windows 2000." Points to an article on employing Uniscribe for formatted layout; also looks at sample code that uses Uniscribe.
  3. The Cssamp.exe file in the Samples subdirectory on the companion CD contains information on Uniscribe coding techniques.


Summary

Uniscribe, a rich technology with regard to typography in multilingual text, offers considerable assistance as you create a world-ready application. Namely, it enables Unicode text to be displayed in all Microsoft-supported scripts and languages, and supplies a script-independent interface for writing applications that display or edit formatted text. In addition, this technology offers simple font fallback for all languages as well as support for OpenType typographic features.

Uniscribe presents a broad range of international support for typographic issues such as kerning. When you use Uniscribe to display text, kerning specified by a font's GPOS table as well as standard ligatures are applied automatically. Uniscribe also provides support for combining marks, which range from the accent marks utilized in Western Europe to the far more complex combinations found in Indic scripts. In addition, it manages bidirectional character reordering using the Unicode bidirectional algorithm. Support for RTL and LTR reading order and for the many-to-many relationship between characters and glyphs are further benefits of this technology. Finally, Uniscribe provides APIs that retrieve the caret position associated with the leading and trailing edges of any character, and that identify the character edge nearest to any pixel offset. For many of the challenges that multilingual text and its typography can present, Uniscribe provides not only a viable but an optimal solution.


Chapter 25: Microsoft Visual Studio .NET

This chapter discusses the benefits of using Visual Studio .NET when creating world-ready applications. Included is an overview of what this development tool provides, where it is available, as well as what international features are associated with it. Finally, the chapter will supply you with additional resources regarding this tool.


Overview of the Tool

Glossary
  • Rapid Application Development: Provides the ability to develop and deploy applications quickly by automating much of the development process and eliminating repetitive tasks.

Visual Studio .NET is the complete development toolset for building Microsoft Windows-based and Web-based applications. There are currently three editions: Microsoft Visual Studio .NET Enterprise Architect, Microsoft Visual Studio .NET Enterprise Developer, and Microsoft Visual Studio .NET Professional. Within the suite of each of these editions are tools that support an integrated development experience, including Microsoft Visual Basic, Microsoft Visual C#, and Microsoft Visual C++. Visual Studio .NET is designed to enhance Rapid Application Development, which allows you to develop Windows and Web applications in an efficient manner and to leverage the possibilities that each environment offers.

The Visual Basic and Visual C# programming languages are based on the Microsoft .NET Framework Common Language Runtime (CLR), which is fully Unicode-enabled. Although there are differences regarding Unicode support among various platforms, CLR hides these differences, so that you don't have to worry about them. You can now create Unicode-based applications without writing extra wrapper code to enable your application for Microsoft Windows 98 and Microsoft Windows Millennium Edition (Me). The .NET Framework CLR is supported on all Windows platforms following Windows 98 and Microsoft Windows NT 4.

Using the Visual Studio development environment, you can create multilingual applications for any language supported in either the .NET Framework CLR or in the Windows operating system. In addition, Visual Studio includes features such as a design-time environment and CLR components that are Unicode-enabled. The Visual Studio editor and the majority of design-time components fully support Unicode on a Unicode-enabled operating system. There are, however, a few components that do not support Unicode, including the Visual C++ Resource Editor.


Availability

Visual Studio can be installed on Windows NT 4, Microsoft Windows 2000, and Microsoft Windows XP. It is localized into eight languages: German, French, Spanish, Italian, Japanese, Korean, Traditional Chinese, and Simplified Chinese. The English version is supported on any of the localized operating systems just mentioned. The localized version is supported on any compatible operating system that has the necessary language support files installed, and on operating systems where the system-locale and user-locale settings match the user interface (UI) language version of Visual Studio.


International Features

Visual Studio has a variety of features that help international teams build and deploy globalized and localized applications. For instance, it offers support for different encodings and for font linking. It is also useful for creating localized Microsoft Windows Forms, localized Web applications, and deployment projects. In addition, Visual C++ handles a variety of character encodings, allows you to use locale-sensitive functions, and provides templates that have their own international features.

Support for Various Encodings and for Font Linking

The Visual Studio editor, which is fully Unicode-enabled, allows you to open and save files in any character encoding supported on the operating system. If you receive a source file that was saved in a particular encoding, you can use the Open With Encoding option, by clicking the drop-down arrow on the Open button, in the File Open dialog box. This dialog box allows you to select any character encoding installed on your system to open your file. You can also use the Advanced Save Options to save your file in a specific encoding.

The Visual Studio editor also supports font linking, which means that your source code can contain multiple scripts; the editor will select the correct font for a particular script. (For more information on font linking, see Chapter 5, "TextInput, Output, and Display.")

Support for Localized Windows Forms

The localization project model in Visual Basic and Visual C# allows you to create applications with multilingual resources and dynamic language switching. Localization projects have a main assembly that holds the actual code for your application, as well as satellite assemblies that contain only the localized resources. These resources can include strings, sizing information, font information, and other properties relevant to localization. (For more information, see Chapter 7, "Soft-ware Localizability Guidelines.")

You can create localized Windows Forms applications within a single project. All Windows Forms projects, whether in Visual Basic or Visual C#, can contain localizable resources. Each Windows Form has a Localizable property that, when set to Tru e, saves all form-based localizable properties in an Extensible Markup Language (XML) resource file with the extension .resx.

You can add new languages to your projects by clicking Language in the Properties window. By selecting a particular language from this window, you can edit and localize a form directly in the Windows Forms Designer. You also have the ability to add strings or other resources by using the Assembly Resource File template and editing the XML-based format in the XML Data View window.

The Assembly Resource File template, which is the XML-based resource file, contains a comment tag that can be used to enter comments for localizers. The comments will be stripped out when the XML resource file is compiled to the binary format. The comment tag is supported only in the Assembly Resource File template, and cannot be used with the Windows Forms Designer.

You can build your main and satellite assemblies from within the integrated development environment (IDE) by simply clicking Build Solution from the Build menu. Satellite assemblies are automatically compiled when the project contains the language-specific .resx files. Satellite assemblies are written to local directoriesunder a path using the culture names based on the Request for Comments (RFC) 3066 standard abbreviations for languages and regions. The format of the culture tags is <language code 2>-<country-region code2 >, where <language code 2> is a lowercase two-letter code derived from International Organization for Standardization (ISO) 639-1, and <country-region code2> is the uppercase two-letter code derived from ISO 3166. A language code is sometimes referred to as a "culture code," and a country-region code is also called a "subculture code." (For more information on culture names and subculture names, see Chapter 4, "Locale and Cultural Awareness." )

The .NET Resource Manager will load localized resources by detecting the UI language of the operating system, unless otherwise specified by the CurrentUICulture property. If resources matching the system UI language are not available, the project will revert to using the resources in the main assembly. (For more information on development within the .NET Framework, go to http://msdn.microsoft.com/net.)

Support for Localized Web Applications

The localization project model in Visual Basic and Visual C# can also be applied to Microsoft Active Server Pages for the .NET Framework (ASP.NET) Web projects, which means you can precompile your main assembly with your application code and use satellite assemblies for your localized resources. (For more information on the ASP.NET application model, go to http://msdn.microsoft.com/net/aspnet.)

You can add and edit localizable resources for your Web projects by using the same Assembly Resource File template that was used for the Windows Forms projects. Visual Studio will automatically generate satellite assemblies if your project contains resource files with a valid language identifier in the file name. To ensure that your resources will be loaded correctly, you must use a valid language identifier in the file name, such as WebApplication1.en-US.resx. (For a list of all valid language identifiers, see Appendix K, "CultureInfo Names and Identifiers in the Microsoft .NET Framework Library.")

You can configure your application through the settings within the XML-based configuration file-Web.config-or through the page directive values. Settings that are relevant to globalization include the following:

  • Culture
  • UICulture
  • RequestEncoding
  • ResponseEncoding
  • FileEncoding

Culture and UICulture are equivalent to the CurrentCulture and Current-UICulture in the System.Globalization namespace. (For more information on CurrentCulture, see Chapter 4.) Culture defines the locale settings for your application, which include currency, date formats, list separators and others. UICulture defines the languages of the resources used in your Web application UI.

Even though all data is processed on the server as Unicode, there are three possible encoding settings for your ASP.NET application. These encodings can either be specified in your application code or in the globalization section contained within each Web.config file. RequestEncoding specifies the encoding the server uses to interpret data sent from the client, and ResponseEncoding determines the encoding of the data that is returned from the server to the client. FileEncoding indicates the encoding ASP.NET will use to interpret the file. If FileEncoding is not set and cannot be determined, ASP.NET will use the Windows code page of the server.

Deployment Projects

Glossary
  • Setup project: Allows you to create installers in order to distribute an application in Visual Studio .NET.

In addition to support for Web projects, Visual Studio allows you to create deployment projects in nine languages, including English, German, French, Italian, Japanese, Korean, Spanish, Simplified Chinese, and Traditional Chinese. Each language version of Visual Studio contains all nine languages. Simply click Localization in the Properties window to create a Setup project with a localized UI. You can develop a single Setup project for each desired language version. Note that Setup projects do not support Unicode characters.

Visual C++ Support for Various Character Encodings

Visual C++ supports source-code files with a non-Unicode character encoding. Extended characters and ideographs can be used in #include file names, comments, character literals, and string literals. Source code can be made portable for compilation on a computer with a different system Windows code page by using #pragma setlocale("<locale string>").

For character-encoding support during run time, Visual C++ includes data types and application programming interfaces (APIs) for handling single-byte character set (SBCS), double-byte character set/multibyte character set (DBCS/MBCS), and Unicode (UTF-16) characters. Macros available in tchar.h allow you to write generic source code that can be compiled for any of these three run-time character encodings.

Locale Model

The Visual C++ run time also provides its own locale schema, which is modeled after the Portable Operating System Interface for UNIX (POSIX) locale model. This allows you to use locale-sensitive functions for string processing, number formatting, currency formatting, date and time formatting, and platform- independent sorting. Similarly, the Standard Template Library (STL) that ships with Visual C++ contains globalization support when C++ templates are used.

MFC and ATL Libraries

A final international feature of Visual Studio involves the libraries of both the Microsoft Foundation Class (MFC) and Active Template Library (ATL), which are available as versions for non-Unicode and Unicode character encoding. MFC and ATL applications that are linked with the Unicode libraries are usually restricted to Windows NT 4 or later. Unicode applications can, however, be statically linked to the Microsoft Layer for Unicode (MSLU) for execution on Microsoft Windows 95, Windows 98, or Windows Me systems. (For more information, go to http://www.microsoft.com/globaldev, or see the Microsoft Windows Platform SDK available at http://msdn.microsoft.com.) The display restrictions for platforms that are not Unicode-based are described in Chapter 3, "Unicode."

The MFC Application wizard contains translated templates for nine languages: English, French, German, Spanish, Italian, Simplified Chinese, Traditional Chinese, Japanese, and Korean. You can select any of these languages for the default UI language of an MFC Application project.

International features for ATL Server include both the possibility of configuring server processing with locale and code-page tags in server response files as well as support for creating Web pages that are encoded in UTF-8.


Additional Resources

For more information on Visual Studio .NET, see the following resources:

  1. http://msdn.microsoft.com/vstudio. Contains detailed documentation for developing international applications with Visual Studio .NET.
  2. http://msdn.microsoft.com/default.asp. Provides material on working in the Visual Studio .NET integrated development environment and using Visual Studio for developing ASP.NET Web Forms. Also contains information on Visual Studio .NET newsgroups and communities, as well as Visual Basic, and Visual C#.


Summary

Visual Studio .NET is a development toolset for building Windows-based and Web-based applications. Some of the tools it includes are Visual Basic, Visual C#, and Visual C++. Visual Studio .NET allows you to create multilingual applications for any language supported in either the .NET Framework run time or in the Windows operating system. Visual Studio also includes features such as a Unicode-enabled design-time environment and Unicode-enabled run-time components.

In terms of international support, with the Visual Studio editor, you can open and save files in any character encoding that is supported on the operating system. The localization project model in Visual Basic and Visual C# allows you to create Windows Forms and ASP.NET Web projects with multilingual resources and dynamic language switching. Moreover, with Visual Studio you can create deployment projects in nine languages. Additional international capabilities include Visual C++ support for various character encodings, and support for locale-sensitive functions such as string processing and sorting, in addition to currency, date, and time formatting. Lastly, ATL Server has international features that allow configuration of server processing with locale and code-page tags in server response files, as well as support for creating Web pages encoded in UTF-8.


Chapter 26: Extensible Markup Language (XML)

This chapter discusses the benefits of using XML when creating world-ready applications. Included is an overview of what this technology provides, where it is available, as well as what international features and international best practices are associated with it. Finally, the chapter will supply you with additional resources regarding this technology.


Overview of the Technology

XML is an open standard for exchanging structured documents and data over the Internet. Introduced by the World Wide Web Consortium (W3C) in November 1996, XML is a simplified version of Standard Generalized Markup Language (SGML). SGML, in turn, was first published by the American National Standards Institute (ANSI) Committee on Information Processing in 1980, though it dates back to the 1960s. SGML has long been used for defining markup languages and for managing large amounts of technical documentation. For instance, IBM, Boeing, and NASA have all utilized SGML for decades. While HTML is an application of SGML that provides a standard way of presenting richly formatted documents in a Web browser, XML is much more general. XML can be used to express literally any structured or semi-structured data and, as a result, has quickly become the lingua franca for data exchange over the Internet. Not surprisingly, given the SGML roots of XML, more and more User Education (UE) teams that produce documentation at Microsoft are switching to XML.

After the introduction of XML in 1996, the W3C subsequently built several core specifications on top of XML, including the following:

  • Namespaces
  • The Document Object Model (DOM)
  • The XML Path Language (XPath)
  • The Extensible Stylesheet Language (XSL)
  • XSL Transformations (XSLT)
  • XML Schema (XSD)
  • XML Digital Signatures (XML Signatures)

In addition, there are other specifications still under development like XML Query, XPointer, XML Encryption, and other XML protocols. (For more information on current and future activities of the W3C with regard to XML-and on XML in general-go to the XML Activity Statement Web site at http://www.w3.org/XML/Activity.)

Microsoft started working in earnest on XML in 1998 when it shipped Microsoft XML Parser (MSXML) in Microsoft Internet Explorer 5. MSXML included support for XML version 1.0, namespaces, the XML DOM, and the precursor to XPath called "XML Query Language" (XQL). XSLT and schema-called "XML-Data Reduced" (XDR)-were also supported.

Since 1998 Microsoft has built XML support into most of its products including Microsoft SQL Server, Microsoft Exchange, Microsoft BizTalk Server, Microsoft Office, the Microsoft .NET Framework, and Microsoft Visual Studio .NET. The .NET Framework includes a complete replacement for MSXML-by offering managed code with full support for XML version 1.0, namespaces, XPath, XSLT, XSD, and XML Signatures.

The demand for internationalization and for a technology that can work across various platforms are the two driving forces behind XML. Having grown out of the Internet, XML is inherently global in nature. But XML has expanded by becoming the "glue" that integrates various programming disciplines. Figure 26-1 shows some high-level areas, the manner in which those areas bridge to XML, and the underlying XML technologies being used to do this. XML can be used in internationalization across all of these areas and more.

Figure 26.1 Programming disciplines that bridge to XML.
Figure 26.1 - Programming disciplines that bridge to XML.

A vivid example of using XML in an internationalization scenario is www.microsoft.com, which reduced approximately 100,000 HTML pages localized in 38 languages down to about 9,000 XML documents. This was accomplished by using Active Server Pages (ASP) code and XSLT stylesheets to generate the HTML pages on the fly, which resulted in a Web site whose production was much easier to manage.

There are some practical limits to using XML that should be mentioned. For example, it is very unlikely that Microsoft will redesign the Microsoft Win32 API to pass XML nodes instead of C Structs, since XML processing is still too CPU-intensive. XML must be used carefully in order to strike the balance between manageability and performance. Tools can also be built in order to post-process XML, which gives a high-performing result. For example, at one point the MSXML XSLT processor was not fast enough, so www.microsoft.com had to post-process their XML and pregenerate all the HTML pages. This situation has changed; live XSLT processing is now possible because computers and the MSXML XSLT process have both gotten a lot faster. Another example of tools to improve performance is in the .NET Framework where XML types can be compiled into C# classes.

Despite the performance limitations just mentioned, XML is having a profound impact upon business practices worldwide for many reasons, among which are its international capabilities (such as its adoption of Unicode as well as other features mentioned in this chapter) and the fact that it is a platform-independent standard. Fortunately, support for processing XML data is also readily available in a wide array of Microsoft products.


Availability

Table 26-1 lists some core Microsoft technologies that offer support for XML. There are too many products that use XML to list them all. (Some of these products are depicted in Figure 26-1.)

Table 26-1 Microsoft technologies that offer XML support.
Product
Description
Internet Explorer 5, Microsoft Windows 2000, Microsoft MSXML
MSXML is a Component Object Model (COM) dynamic-link library (DLL) containing an XML parser and DOM. Version 2 shipped in Internet Explorer 5, providing XML support to the browser in the form of XML data islands, XML data binding, and a built-in XML viewer using XSLT. Version 2.5 became part of the Windows 2000 platform and shipped in Microsoft Windows 98 Service Pack 2 (SP2) and Windows Millennium Edition (Me). Version 3, which has been developed further with Simple API for XML (SAX) and XSLT, is included in Microsoft Internet Explorer 6 and Microsoft Windows XP. MSXML 4 includes XSD support and is available for download on MSDN. (For more information, go to http://msdn.microsoft.com/xml)
Microsoft SQL Server 2000
SQL Server 2000 has built-in support for XML. It provides Open XML for XML support from within stored procedures. It also supplies Internet Information Services (IIS) extensions, so you can query or update the database either by using special URLs or by posting special XML-based queries and updates.
Simple Object Access Protocol (SOAP) Toolkit
The SOAP toolkit was first developed and shipped as a software developer kit (SDK) on the Microsoft Developer Network (MSDN). Since that time, SOAP extensions for reliable messaging have been built into BizTalk Server and Microsoft Message Queuing (MSMQ) 3.
Microsoft Exchange 2000
Exchange 2000 supports Web Distributed Authoring and Versioning (DAV), which is an Internet Engineering Task Force (IETF) XML standard. Exchange 2000 uses the MSXML parser mentioned previously.
.NET Framework
XML is a core part of the .NET Framework as shown by the following areas:
  • The System.Xml namespace contains the core XML classes that support XML 1.0, DTDs, XDR, XSD, XPath, XSLT, and the DOM; it also has a new efficient way of reading and writing XML at the stream level (XmlReader/XmlWriter).
  • The System.Xml.Serialization namespace together with a tool called "xsd.exe" provides support for compiling XML types into classes and then streaming XML into and out of those objects.
  • The System.Data namespace contains some new XML-to-relational mapping capabilities in DataSet in conjunction with XmlDataDocument.
  • System.Runtime.Serialization supports SOAP-based object remoting.
  • System.Web.Services is fully XML-based.
  • System.Resources contains a new XML resource format-.resx.
  • System.Security uses an XML format for persisting custom security policies.
  • System.Security.Cryptography.Xml provides support for XML Digital Signatures.
  • System.Configuration manages .config files that are in an XML format.
Visual Studio .NET
C# itself supports a new XML-based commenting feature for providing program-level reference documentation. Visual Studio .NET provides a new IntelliSense-aware XML editor, and the XML Designer can generate XSD schemas. The new Visual Studio project files are also in an XML format. (For more information, go tohttp://msdn.microsoft.com/vstudio.)
Universal Description, Discovery, and Integration (UDDI)
UDDI is a new Web service provided by Microsoft and IBM that contains a registry for Web services; this enables programs to bind to Web-based services dynamically. (For more information, see http://uddi.microsoft.com.)
Microsoft Office XP
All versions of Office XP, Microsoft Excel, and Microsoft Access have new native support for importing and exporting XML data.


International Features

In addition to the fact that XML is readily available, its international features make it even more compelling in the creation of world-ready applications. Some of XML's multiple international features include encoding declaration and character entities. Encoding information can also be contained in HTTP headers.

Encoding Declaration

XML is a text format and uses mechanisms similar to those of HTML in terms of specifying character encodings. In XML, an optional encoding attribute on the XML declaration defines the character encoding. For example, the following encoding declaration indicates that the encoding is International Organization for Standardization (ISO) 8859-1:

<?xml version='1.0' encoding='ISO-8859-1'?>

To determine the default encodings, a specific algorithm is used. If the file starts with a 4-byte Unicode byte-order mark [0xFF 0xFE 0xFF 0xFE] or [0xFE 0xFF 0xFE 0xFF], the document is considered to be in UTF-32 encoding. If the file starts with a 2-byte Unicode byte-order mark [0xFF 0xFE] or [0xFE 0xFF], the document is considered to be in UTF-16 encoding. Otherwise, the character encoding of the document defaults to UTF-8. This default can only be changed by the presence of an XML declaration or of a Content-Type HTTP header. Table 26-2 shows different encodings of the same XML data, as well as the HTTP headers that are used for each encoding.

Table 26-2 Encodings and HTTP headers used for XML data.
Character Set or Encoding | HTTP Header | XML Document
ISO 8859-1 | Content-Type: text/xml; charset:ISO 8859-1; | <test> </test>
UTF-8 | Content-Type: text/xml; | <test>ã¥</test>
UTF-8 with UTF-8 byte-order mark | Content-Type: text/xml; | ﻿<test>ã¥</test>
ISO 8859-1 | Content-Type: text/xml; | <?xml version="1.0" encoding="ISO 8859-1"?> <test> </test>
UTF-8 (using character entities) | Content-Type: text/xml; | <test>&#229;</test>
UTF-16 (Unicode with byte-order mark) | Content-Type: text/xml; | ff fe 3c 00 74 00 65 00 73 00 74 00 3e 00 e5 00 ..<.t.e.s.t.>... 3c 00 2f 00 74 00 65 00 73 00 74 00 3e 00 0d 00 <./.t.e.s.t.>... 0a 00

See Appendix J, "Encoding Web Documents," for more XML character set or encoding labels.

Character Entities

Like HTML, XML allows individual characters in a page to be encoded by specifying their exact Unicode character value. These character entities are then parsed independently of the character set, and their Unicode values can be determined unambiguously. These values can be specified in decimal format (&#229;) or in hexadecimal format (&#xE5;).

HTTP Headers

HTTP headers can contain character-encoding information in the Content-Type Header, which MSXML and the .NET System.Xml.XmlTextReader can use. For example, the following HTTP header shows the character encoding, in bold.

HTTP/1.1 200 OK
Content-Length: 15327
Content-Type: text/html; charset:ISO 8859-1;
Server: Microsoft-IIS/5.0
Content-Location: http://www.microsoft.com/default.htm
Date: Wed, 08 Dec 1999 00:55:26 GMT
Last-Modified: Mon, 06 Dec 1999 22:56:30 GMT

<test>This is some XML encoded in the ISO 8859-1 character set</test>

Even though the XML in the message has no XML encoding declaration, the HTTP header indicates that the XML content is in the ISO 8859-1 character set.


International Best Practices

The following contains best practices when working with XML in the global environment. Since it is common to encounter character-encoding issues when first learning XML and dealing with MSXML, you will see pointers and examples of what to watch for as you use MSXML.

Use UTF-8

The default encoding for all XML documents (or messages) is UTF-8, which is a compressed encoding of Unicode (ISO 10646). If possible, stick with UTF-8, especially if you want to maximize cross-platform reach.

Make Sure Your XML Data Is Locale- and Culture-Neutral

Remember that using XML entails making sure your data is cleanly separated from your user interfaces (UIs). Therefore, follow the W3C XML Schema standard and note how you store your data in XML. For example, if you store your dates in ISO 8601 format, the date May 17, 2001 would be represented as 2001-05-17. The application that uses XML must decide how to present data to the user based on the user's regional settings. (For more information on adjusting data according to the user's regional or language settings, see Chapter 4, "Locale and Cultural Awareness." )

Take Advantage of MSXML Tips and Tricks

If you are transmitting data using XML, it is recommended that you use the default UTF-8 encoding. However, the following sample shows how to transmit data reliably, between client and server, if you have to use a different encoding. Suppose you have the following XML file on your server:

<?xml version='1.0' encoding='windows-1252'?>
<test>Copyright © 2001</test>
Note
This code represents the simplest way to encode the copyright symbol because the symbol has a single-byte representation in the Windows 1252 character set.

If the XML file were encoded in UTF-8, the raw file would look like this:

<?xml version='1.0' encoding='utf-8'?>
<test>Copyright © 2001</test>
Note
Notepad in Windows 2000 or Windows XP supports UTF-8 and will decode the "©" sequence, so you won't see the raw format. If you type the file from a Microsoft MS-DOS command window, you will see this raw UTF-8 encoding.

In the previous code, the "©" sequence is a two-byte UTF-8 encoding of the Unicode character 0xA9. To send this XML document from an ASP page, therefore, you would do the following:

<%@LANGUAGE=JScript%>
<%
      Response.ContentType = "text/xml";
      Response.Clear();
      var dom = Server.CreateObject("MSXML2.DOMDocument");
      dom.load(Server.MapPath("test.xml"));
      dom.save(Response);
%>
Note
This code uses the MSXML 3 class ID "MSXML2.DOMDocument." The MSXML 2 version is called "Microsoft.XMLDOM."

To receive the XML document on the client using the XMLHTTP component in Microsoft JScript, you would do the following:

var http = new ActiveXObject("MSXML2.XMLHTTP");
http.open("GET","http://localhost/test.asp",false);
http.send();
doc = http.responseXML;
doc.save("test.xml");

To execute this script, run the following from the command line:

cscript /nologo test.js

This will create a file on your local directory called "test.xml," which is correctly encoded in the Windows 1252 character set. Be careful using WScript.echo with XML content. For example, if you add the following to the JScript code:

WScript.echo (doc.xml);

You will see the following incorrect output:

<?xml version="1.0"?>

<test>c</test>

This is because WScript.echo is losing information in the translation of Unicode (doc.xml) to the console character set. Also, notice that the xml property on the DOM document object removed the encoding attribute. This is by design. The xml property is returning a correctly encoded Unicode version of your XML document that can then be successfully parsed by LoadXML.

Therefore, the two main methods of loading XML documents using the MSXML DOM are the LoadXML method and the Load method. The LoadXML method always takes a Unicode BSTR that is encoded in UCS-2 or UTF-16 only. If you pass in anything other than a valid Unicode BSTR to LoadXML, it will fail to load. The Load method takes a VARIANT and supports the following values, shown in Table 26-3.

Table 26-3 Values supported by the Load method.
Value | Description
URL | If the VARIANT is a BSTR, it is interpreted as a URI, and the XML content is fetched from that URI.
VT_ARRAY | VT_UI1 | The VARIANT can also be a SAFEARRAY containing the raw encoded bytes.
IUnknown | If the VARIANT is an IUnknown interface, the DOM document calls QueryInterface for IStream, IPersistStream, and IPersistStreamInit.

The Load method implements the following algorithm for determining the character encoding or character set of the XML document:

  1. If the Content-Type HTTP header defines a character set, this character set overrides anything in the XML document itself. This obviously doesn't apply to SAFEARRAY and IStream mechanisms because there is no HTTP header.
  2. If there is a 2-byte Unicode byte-order mark, the Load method assumes the encoding is UTF-16. It can handle both big-endian and little-endian.
  3. If there is a 4-byte Unicode byte-order mark (0xFF 0xFE 0xFF 0xFE), the Load method assumes the encoding is UTF-32. It can handle both big-endian and little-endian.
  4. Otherwise, the Load method assumes the encoding is UTF-8, unless it finds an XML declaration with an encoding attribute that specifies some other character set (such as ISO 8859-1, Windows 1252, Shift-JIS, and so on).

There are two errors you will see returned from the MSXML DOM that indicate encoding problems. The first usually indicates that a character in the document does not match the encoding of the XML document:

An invalid character was found in text content.

The ParseError object will tell you exactly where on a particular line this rogue character occurs so that you can fix the problem. The second error indicates that you started with a Unicode byte-order mark (or that you called the LoadXML method) and that an encoding attribute specified something other than a 2-byte encoding (such as UTF-8 or Windows 1250):

Switch from current encoding to specified encoding not supported.

Another possibility is that you could have called the Load method and started off with a single-byte encoding (no byte-order mark), but the method might have found an encoding attribute that specified a 2- or 4-byte encoding (such as UTF-16 or UTF-32).

Note
The LoadXML method is more forgiving in MSXML 3. It allows any XML declaration to be passed in, but still assumes the real encoding is Unicode (since a BSTR is always Unicode by definition).

Finally, the IXMLHttpRequest interface provides several ways of accessing downloaded data, shown in Table 26-4.

Table 26-4 Properties of the IXMLHttpRequest interface, used to access downloaded data.
Properties | Description
ResponseXML | Represents the response entity body as parsed by the MSXML DOM parser, using the same rules as the Load method.
ResponseText | Represents the response entity body as a string.
ResponseBody | Represents the response entity body as an array of unsigned bytes.
ResponseStream | Represents the response entity body as an IStream interface.
Note
A common problem people experience involves the ResponseText property. This property assumes the message is in UTF-8 encoding. Consequently, if you send a message that is in another encoding, the property will not work correctly.

Take Advantage of System.Xml Tips and Tricks

While Chapter 19, "The Microsoft .NET Framework," provides an overview of the .NET Framework, this section offers more detail on the System.Xml classes within the framework. It looks specifically at the same scenarios just discussed for MSXML.

The System.Xml classes are built upon lower-level classes that handle networking (such as System.Net) and encoding/decoding text (such as System.Text). The main difference between System.Xml and MSXML is that, instead of using the XMLHTTP control for HTTP client code to send and receive XML data over the Internet, you use the classes defined in the System.Net namespace, which are far more powerful and flexible. Similar to MSXML, the System.Xml classes allow you to load XML documents from in-memory strings encoded in UTF-16, or from encoded streams where the decoding rules are the same. In the .NET Framework you can also load XML documents from a text reader, which is a stream that has already been decoded into Unicode. The following is the managed Microsoft ASP.NET (once known as "ASPX") equivalent of the earlier ASP server-side code fragment used in the MSXML scenario:

<%@LANGUAGE=C#%>
<%@Import Namespace="System.Xml"%>
<%
   Response.ContentType = "text/xml";
   Response.Clear();
   XmlDocument doc = new XmlDocument();
   doc.Load(Server.MapPath("test.xml"));
   doc.Save(Response.Output);
%>

This will send a UTF-8-encoded response to the client. The following is the simplest managed client-side code for downloading and decoding this XML data correctly:

using System.Xml;
using System.Net;
class Test
{
   static void Main() {
      XmlDocument doc = new XmlDocument();
      doc.Load("http://localhost/temp/test.aspx");
      doc.Save("saved.xml");
   }
}

If you need more control over the specific HTTP client-side code, use the System.Net.WebRequest class. For example, the following code adds the Network-Credentials class to download data from a Web site that requires authenticated access.

using System.Xml;
using System.Net;
class Test
{
   static void Main() {
      WebRequest wr =
         WebRequest.Create("http://localhost/temp/test.aspx");
      wr.Credentials = CredentialCache.DefaultCredentials;
      XmlDocument doc = new XmlDocument();
      doc.Load(wr.GetResponse().GetResponseStream());
      doc.Save("saved.xml");
   }
}

This will retrieve the UTF-8-encoded response and generate a UTF-8-encoded local file named "saved.xml."

There is a tricky issue in this particular ASP.NET scenario. ASP.NET picks up the "default" encoding-UTF-8-for the HTTP response from the Web.Config file. Therefore, do not set the Response.Charset property to something other than UTF-8. If you do, you will get an invalid response because the XML body will be encoded as UTF-8, but the HTTP Content-Type header will say the charset is something else. As a result, MSXML and Internet Explorer will be confused, since they will attempt to decode the body according to the charset in the HTTP header, and not according to UTF-8, resulting in a garbled XML message.

If you must respond with something other than the UTF-8 encoding, change the Web.Config file as follows:

<configuration>
   <system.web>
      <globalization
         responseEncoding="iso-8859-1" />
   </system.web>
</configuration>

This code will now cause the ASP.NET page to respond with the XML body correctly encoded in the ISO 8859-1 encoding. If you don't want to or cannot change the Web.Config file for some reason, you can use the following code instead:

<%@LANGUAGE=C#%>
<%@Import Namespace="System.Xml"%>
<%
   Response.ContentType = "text/text";
   Response.Charset = "iso-8859-1";
   Response.Clear();
   XmlDocument doc = new XmlDocument();
   doc.Load(Server.MapPath("test.xml"));
   XmlTextWriter xw = new XmlTextWriter(
                         Response.OutputStream,
                         Encoding.GetEncoding(Response.Charset));
   xw.Formatting = Formatting.Indented;
   doc.Save(xw);
   w.Close();
%>

This code makes sure the XML body is encoded according to the Charset property. The client-side code (both the MSXML and System.Xml versions) will decode the XML body correctly, and the local file called "saved.xml" will preserve the ISO 8859-1 encoding.


Additional Resources

For more information on XML, see the following resources:

  1. http://msdn.microsoft.com/xml . Serves as a general reference; the Microsoft XML Developer Center Web site.
  2. http://www.w3.org . Points to reference material; the World Wide Web Consortium's Web site, where XML standards officially reside.
  3. http://www.xml.org . Shows various vertical-industry, XML standardization efforts.
  4. http://msdn.microsoft.com/xml/articles/xml09182000.asp . Lovett, Chris. "XML and www.microsoft.com." September 2000. Discusses how www.microsoft.com uses XML.
  5. http://msdn.microsoft.com/xml/articles/xmlencodings.asp .Lovett, Chris. "How to Encode XML Data." March 2000. Explains how character encoding works, particularly within XML and the MSXML DOM.


Summary

XML is a standard for exchanging structured documents and data over the Internet. Unlike HTML, which defines a specific document format, XML is much more generalized and can represent any data, just as SGML can. The demand for internationalization and for a technology that can work across various platforms in a standard way are the two driving forces behind XML. Having grown out of the Internet, XML is inherently global in nature, making it a powerful technology to use in creating world-ready applications. XML technologies are now integrated with various programming areas such as relational databases, distributed programming, document and page processing, as well as messaging and Web services.

One of the international features that XML comprises is encoding declaration, whereby an optional encoding attribute on the XML declaration defines the character encoding. A second international feature is related to character entities-like HTML, XML allows individual characters in a page to be encoded by specifying their exact Unicode character value. Indeed, XML's potential for improving business practices worldwide and for fostering the development of world-ready applications is easily apparent.

As you work with XML, try to use UTF-8 or UTF-16 encoding in order to reach as many platforms as possible. Make sure your data is language-neutral. Also, follow the W3C XML Schema standard and note how you store your data in XML. As a final best practice, take advantage of the many MSXML tips and tricks that have been outlined here and keep up to date with the new XML features of the .NET Framework.


Glossary

.NET Framework A platform that enables the creation and use of Extensible Markup Language (XML)-enabled applications, processes, and Web sites as services that share and combine information and functionality with each other by design. The .NET Framework is available for Windows platforms or smart devices, providing tailored solutions for organizations and individuals. See Common Language Runtime (CLR).

-A APIs The Win32 application programming interface (API) entry points that usually expect string parameters to be encoded in Windows code pages (also called "ANSI code pages") or sometimes OEM code pages. See -W APIs.

Accelerator An Alt+character combination used to activate menus, menu items, and dialog box items in Windows. The character that activates the menu or dialog box item is underlined. It is also called a "hot key."

Accented character A character that has a diacritic attached to it-such as "ñ." See Extended characters and Diacritic.

Accessibility The extent to which computers are easy to use and available to a wide range of users, including people with disabilities.

ACP Acronym for the Windows (ANSI) code page in use. Windows NT uses this code page to convert to and from Unicode (UTF-16) automatically whenever an application calls one of the "A" entry points of Win32 APIs.

Active Input Method Manager (IMM) An ActiveX control that provides limited IMM service on non-Asian language versions of Windows 95, Windows 98, Windows Me, and Windows NT 4 platforms. It is replaced by the more general Text Services Framework in Windows XP. Active IMM is also known as "Global IME."

ADO.NET Stands for Microsoft ActiveX Data Objects for the .NET Framework.A set of classes that expose data-access services to the .NET programmer. ADO.NET supplies a rich set of components for creating distributed, data-sharing applications. It is an integral part of the .NET Framework, providing access to relational data, XML integration, and application data.

Alphabet Elements of a writing system composed of a collection of letters that have a one-to-one relationship with a sound. See Letter.

Alphanumeric Consisting of either letters or numbers, or both.

Alternates In fonts, alternates are similar to positional forms. For example, kanji uses alternative forms of parentheses when positioned vertically.

AltGr The Alt key on the right on some non-U.S. Windows keyboard layouts. The AltGr key is equivalent to the Ctrl+Alt key combination and is used to create an alternative shift state for accessing additional characters on some keys.

Alt+Numpad A method of entering characters by typing in the character's decimal code with the Numeric Pad keys (Num Lock turned on). In Windows:

  • Alt+<xxx>, where xxx is the decimal value of a code point, generates an OEM-encoded character.
  • Alt+<0xxx>, where xxx is the decimal value of a code point, generates a Windows-encoded character.
  • Alt<+>+<xxxx>, where xxxx is the hexadecimal Unicode code point, generates a Unicode-encoded (UTF-16) character.

ANSI Acronym for the American National Standards Institute. The term "ANSI" as used to signify Windows code pages is a historical reference, but is nowadays a misnomer that continues to persist in the Windows community. The source of this comes from the fact that the Windows code page 1252 was originally based on an ANSI draft-which became International Organization for Standardization (ISO) Standard 8859-1. "ANSI applications" are usually a reference to non-Unicode or code page-based applications.

ANSI C The standardized C programming language.

Anti-aliasing A software technique for smoothing the jagged appearance of curved or diagonal lines caused by poor resolution on a display screen.

Application programming interface (API) A set of functions supported by the operating system.

ASCII Acronym for American Standard Code for Information Interchange, a 7-bit encoding. Although primitive, ASCII's set of 128 characters is the one common denominator contained in most of the other standard character sets and in all Windows and OEM code pages.

ASP.NET Stands for Microsoft Active Server Pages for the .NET Framework. The new generation of Active Server Pages (ASP) files written in a managed language on the Common Language Runtime (CLR) using the .NET Framework. Also known as "ASP+," and "ASPX."

Attribute (1) In C/C++ programming, attributes contain information or parameters that define object classes property and/or behaviors. (2) In HTML, an attribute is a parameter that defines a special property of an HTML element. Attributes are specified within start tags. For example, <IMG SRC="image.gif">means that the elementIMG has an attribute called "SRC," which is assigned the indicated value. (3) In XML, an attribute is similar to HTML attributes, but describes additional properties on any XML element.

Base character An encoding code point that does not graphically combine with preceding characters and that is neither a control nor a format character. The Latin "a" is an example of a base character.

Beta testing Distributing prerelease software to future users and potential customers in order to get feedback and bug reports.

Bidirectional (BiDi) rendering Refers to the script's ability to handle text that reads both left to right and right to left. For example, in the bidirectional rendering of Arabic, the default reading direction for text is right-to-left, but for numbers, it is left-to-right. Processing a complex script must account for the difference between the logical (keystroke) order of input and the visual order of the output glyphs. In addition, processing must properly deal with caret movement and hit testing. The mapping between screen position and a character index for, say, selection of text or caret display requires knowledge of the layout algorithms.

Bidirectional text A mixture of characters that are read from left to right and from right to left. Most Arabic and Hebrew strings of text for example, are read from right to left, but numbers and embedded Western terms within Arabic or Hebrew text are read from left to right.

Big-5 The multibyte encoding for Traditional Chinese characters standardized by Taiwan.

Big-endian A computer architecture that stores multibyte numerical values with the most significant byte values first. On systems using big-endian architecture, the letter "A" (U+0041) is stored as 0x00 0x41. See Little-endian.

Binary file A file that has been encrypted, encoded, or compiled, as opposed to a plaintext file.

Bitmap font A font whose characters are represented by bitmaps or by a pattern of dots, as opposed to vector, TrueType, or OpenType fonts, whose characters are represented by lines and curves. Bitmap fonts are generally less scalable and more jagged than TrueType fonts. See TrueType.

Bopomofo A set of characters used to teach the phonetics of Chinese. These characters are used in teaching materials such as dictionaries, but do not appear in the actual writing of the Chinese language.

Boundary The point of interaction between systems or applications that use different character encodings.

BSTR Known as a "basic string" or "binary string." A BSTR is a pointer to a wide-character string used by Automation data-manipulation functions. BSTR strings are also used for passing string data between Component Object Model (COM) components, by utilizing strings allocated with COM's memory allocator.

Byte-order mark (BOM) The Unicode character U+FEFF-or its noncharacter mirror-image, U+FFFE-used to indicate the byte order of a text stream. The presence of a BOM at the beginning of a text file is a strong clue that a file is encoded in Unicode.

Candidate window The window of an Input Method Editor (IME) that lists characters the user can choose to replace the text highlighted in the composition window.

Cardinal splines In GDI+, a sequence of individual curves joined to form a larger curve.

Caret The blinking line indicating the space into which you insert text.

Case Two distinct variations or forms of the same character within the same alphabet. These variants, which differ in shape and size, are called "uppercase" letters (also known as "capital" or "majuscule" letters) and "lowercase" letters (also known as "small" or "minuscule" letters).

Case-folding Taking a string of text and converting everything into either lowercase or uppercase.

Chang Jei An input method that uses radicals to build Chinese characters. See Radicals. Twenty-five radicals are assigned to the letters "A"through "Y. " The letter "X"is used to generate more complex radicals. See Input Method Editor (IME).

Character (1) The smallest components of a writing system or script that have semantic value. A character refers to an abstract idea rather than to a specific glyph or shape that a character might have once rendered or displayed. (2) A code element.

Charset Stands for "character set." A set of characters used in Windows. Charsets refer to the same collections of characters as those defined by Windows code pages. See Code page.

CJK/CJKV A reference to Chinese, Japanese, and Korean languages. Sometimes a "V" for Vietnamese is added to the acronym. See Multibyte character set (MBCS) and Double-byte character set (DBCS).

Client coordinates Relative coordinates of a window or client area as specified by the system or applications. Client coordinates ensure that an application can use consistent coordinate values while drawing in the window, regardless of the position of the window on the screen.

Clipboard A Windows utility used as a buffer for copying and pasting text.

Clusters The sequence of characters or glyphs between points at which the Unicode representation of a string aligns with the glyph representation. For simple text, where each code point is represented by a single glyph, the cluster is the character and its glyph. For simple ligatures, where two or more code points are represented by a single glyph, the cluster is the sequence of code points and the single glyph. The most complex case is when a sequence of code points is represented by a sequence of glyphs with no internal alignment between characters and glyphs. This can occur, for example, in the case of reordering within Indic syllables.

Code page An ordered set of characters of a given script in which a numeric index (code-point value) is associated with each character. In this book, this term is generally used in the context of code pages defined by Windows and can also be called a "character set" or "charset."

Code point, or code element (1) The minimum bit combination that can represent a unit of encoded text for processing or exchange. (2) An index into a code page or a Unicode standard.

Collation Refers to a set of rules that determine how textual data is sorted and compared.

Combining character A character that graphically combines with a preceding base character. The combining character is said to apply to that base character. The combining acute accent mark (h)-U+0301-is an example of a combining character.

Combining-mark sequence An alphabetic base character followed by one or more combining-mark characters such as acute and grave accents.

Combined characters or ligatures Characters that join into one character when placed together. One example is the "ae" combination in English; it is sometimes represented by a single character. Arabic is a script that has many combining characters.

Commenting model A way for the development team to pass along localization instructions to the localization team. This information might include whether strings can be localized or whether limitations on the string length exist, for instance, or can just consist of general comments added to the resource files. See localization.

Common controls Within the context of Windows, a set of controlling elements (windows) that are implemented by the common control library, which is a dynamic-link library (DLL) included with the Windows operating system. Like other control windows, a common control is a child window that an application uses in conjunction with another window to perform input/output (I/O) tasks.

Common dialog boxes Standard dialog boxes defined by Windows for operations found in numerous applications; these operations include Open, Save As, Print, Page Setup, Color Selection, Font Selection, and Find. Applications can call common dialog box API functions directly instead of having to supply a custom dialog template and dialog procedure.

Common Language Runtime (CLR) A very important part of the .NET Framework. At the base level, it is the infrastructure that executes applications and allows them to interact with the other parts of the .NET Framework. It also provides important capabilities in optimizing, securing, and providing many robust capabilities such as application deployment and side-by-side execution. See .NET Framework.

Compatibility zone The area in the Unicode repertoire from U+F900 through U+FFEF that is assigned to characters from other standards. These characters are variants of other Unicode characters.

Complex scripts Scripts that require special handling when it comes to shaping and laying out characters in software applications. This special handling is closely related to linguistic requirements of these scripts. Complex scripts can have any combination of the following attributes: bidirectional rendering, contextual shaping, combining characters, as well as specialized word-breaking and justification rules. See Contextual analysis, Rendering, and Uniscribe.

Component Object Model (COM) A specification that Microsoft developed for building software components that can be assembled into programs or that add functionality to existing programs running on Microsoft Windows platforms.

Console The Windows subsystem that runs character-based applications, as opposed to applications that have a graphical user interface (GUI).

Constant A numeric value, typically an integer, that refers to a character value, the size of a buffer, the position of a character in a string, and so forth. It is assumed that the value does not change during the time a program is running.

Content recycling Reusing content in localization. Content recycling saves time and money because the content is researched, presented, edited, reviewed, and translated only once. Translation memory is a common form of content recycling.

Contextual analysis A process for determining how to handle text based on surrounding characters, as in Arabic, in which a glyph changes shape depending on its position in a word. See Complex scripts, Rendering, and Uniscribe.

Control Panel A group of Windows utilities used to edit system settings, including international preferences.

Conversion or composition window The window of an IME that displays text typed by the user, either just the way it is entered or after it is converted to ideographic form.

Cross-platform Portable or applicable to more than one operating system.

Cultural convention Data or data formats that are specific to a language, local dialect, or geographic location. Examples are currency symbols, date formats, calendars, numeric separators, and sort orders.

Cursive attachment Used when adjacent glyphs need to be positioned in order to join them cursively. It is heavily utilized in fonts that support cursive scripts like Arabic. See Kashida.

Cyrillic script The script traditionally used for writing various Slavic languages, including Russian. Over the past two centuries, the Cyrillic script has been extended so that some of the other non-Slavic minority languages of the former Soviet Union could be written. Cyrillic script is written in linear sequence from left to right.

Date picture string/time picture string A string used to represent a date or time format-for example, "dd MMMM, yyyy".

Dead key A key that does not produce a character by itself, such as the accent key on the international keyboard. However, when the user types in a character after pressing the accent key, an accented character appears.

Decomposition The breakdown of an accented character or a precomposed character into an ordered set of character components. For "ã" the components are "a" followed by the combining character "~ ".

Determined string A string that has been converted from a phonetic representation into ideographs.

Device context A Graphics Device Interface (GDI) structure that defines a set of graphics objects and their associated attributes, as well as the graphics modes that affect output. The graphics objects include a pen for line drawing, a brush for painting and filling, a bitmap for copying or scrolling parts of the screen, a palette for defining the set of available colors, a region for clipping and other operations, and a path for painting and drawing operations.

Device Driver Kit (DDK) A set of tools and libraries for creating Windows-based software to run hardware devices such as printers, along with documentation.

Diacritic A character that is attached to or overlays a preceding base character. For example, a mark placed over, under, or through a Latin-based character-such as "~"-to indicate a change in phonetic value from the unmarked state. Most diacritics are nonspacing characters that don't increase the width of the base character. See Accented character.

Diaeresis Two dots placed over a Latin vowel to indicate that the vowel is pronounced as a separate syllable (as in the word "naïve"). Typically used when two vowels are adjacent, but should be pronounced separately rather than as a diphthong. See Umlaut.

Digital Dashboard A container that provides a customized display of information consolidated from various information sources. Every Digital Dashboard is a Web page containing one or more Web Parts. See Web Parts.

Digraph A combination of characters that is written separately but forms a single lexical unit-for example, the Danish "aa" and the Spanish "ch"and "ll".

Double-byte character set (DBCS) A character encoding in which the code points can be either 1 or 2 bytes. Used, for example, to encode Chinese, Japanese, and Korean languages. See Multibyte character set (MBCS), and CJK/CJKV.

Dynamic-link library (DLL) A module containing functions or resources that other programs or DLLs can utilize. DLLs cannot run by themselves; other programs have to load them.

EBCDIC Extended Binary Coded Decimal Interchange Code. These types of code pages are used on IBM and other manufacturers' mainframes.

Element The basic unit of information in an HTML or XML document. Elements are arranged hierarchically to define the overall document structure.

Enabling Altering program code to handle input, display, and editing of bidirectional or East Asian languages, such as Arabic and Japanese, respectively.

Encoding A method or system of assigning numeric values to characters (for example, ASCII, Unicode, Windows 1252).

End-User Defined Character (EUDC) A special character, such as a rare ideograph, that the user creates with a EUDC editor and assigns to a code point within a reserved range.

Extended characters (1) Characters above the ASCII range (32 through 127) in single-byte character sets. (2) Accented characters.

Extensible Stylesheet Language (XSL) An XML language used for transforming XML documents into something that can be displayed, such as HTML.

Floating accent See Diacritic and Floating diacritic.

Floating diacritic A combining character that overlays the preceding base character; it can potentially change position or shape according to the shape of the base character. The combining right arrow above-character (U+20D7) is an example of a floating diacritic.

Following characters Characters-such as closing quotation marks, closing parentheses, and punctuation marks-that shouldn't be separated from succeeding characters.

Font Any of numerous sets of graphical representations of characters that can be installed on a computer, printer, or another graphic output device.

Font association The automatic pairing of a font that contains ideographs with a font that does not contain ideographs. This allows the user to enter ideographic characters regardless of which font is selected.

Font fallback Mechanism for providing an alternate font for runs of characters not representable by the original font. Understands complex scripts and typographic effects. Font fallback is a hard-coded list of standard fallback fonts according to the Unicode character range.

Font linking Mechanism for linking missing glyphs to the currently selected font (also called "base font") from alternative fonts (also called "linked font"). See Font association and Font fallback.

Font substitution The explicit replacement of any reference or call to a given font face name with another face name. For example, MS Shell Dlg is only a face name and has no associated physical font. It is substituted by Microsoft Sans Serif in the English version of Windows XP, and varies per localized version.

Front-end processor See Input Method Editor (IME).

Full-width character Characters whose glyph image extends across the entire character display cell. In legacy character sets, full-width characters are normally encoded in 2 or more bytes. The Japanese term for full-width characters is "zenkaku." See Half-width character.

GB 2312-80 A multibyte encoding standardized by the People's Republic of China.

Generic data type A macro, such as TCHAR, that resolves to either an ANSI type or a wide-character (Unicode) type, depending on compile-time flags.

Generic prototype A macro representing an API call or a function call. The macro resolves to an entry point that expects either ANSI parameters or wide-character (Unicode) parameters, depending on compile-time flags.

Global.asa A file typically containing scripts that initialize application or session variables, connect to databases, send cookies, and perform other operations pertaining to the ASP application or to the user's session with an ASP application as a whole.

Globalization The process of developing a program core whose features and code design are not solely based on a single language or locale. Instead, their design is developed for the input, display, and output of a defined set of Unicode-supported language scripts and data related to specific locales. See Internationalization.

Globalized functionality testing Functionality testing that has been enhanced to include verifying the world-readiness of a product.

G11N Abbreviated form of GLOBALIZATION. G + 11 characters + N.

Glyph The actual shape (bit pattern, outline, and so forth) of a character image. For example, an italic "a " and a roman "a" are two different glyphs representing the same character.

GMT Greenwich Mean Time. See UTC.

Graphics Device Interface (GDI) In Windows, a graphics display system used by applications to display or print bitmapped text (TrueType fonts), images, and other graphic elements. The GDI, in particular, is responsible for drawing dialog boxes, buttons, and other elements.

Gregorian calendar A solar-based dating system used as the default calendar of countries in the Western hemisphere and also used widely in other parts of the world.

Group Policy Centralized policy-based administration that enables an administrator to control or specify registry-based policy settings, security settings, software installation, scripts to run at computer startup and operating-system shutdown, Internet Explorer maintenance, and folder redirection. Some group policy features may require the installation of Active Directory to work.

Half-width character Characters whose glyph image occupies half of the character display cell. In legacy character sets, half-width characters are normally encoded in a single byte. The Japanese term for half-width characters is "hankaku." See Full-width character.

Han unification The process of assigning the same code point to characters historically perceived as being the same character but represented as unique in more than one East Asian ideographic character standard. This results in a group of ideographs shared by several cultures and significantly reduces the number of code points needed to encode them.

Hangul The native name for the Korean language.

Hanja The Korean name for ideographic characters of Chinese origin.

Hanzi (hantsu) The Chinese name for ideographic characters of Chinese origin.

Hard-coding (1) Putting string or character literals in the main body of code, such as the .C files or the .H files, instead of in external resource files. (2) Basing numeric constants on the assumed length of a string or having any assumptions about language- or culture-specific matters fixed in the code (such as length of strings, formats of dates, and so on).

Hebrew lunar calendar A calendar based on the cycle of the moon around the Earth. The length of this cycle, the lunar month, is about 29½ days. Twelve lunar months make, therefore, about 354 days.

Hijri (Islamic lunar) calendar Since the Islamic calendar is purely lunar, as opposed to solar or luni-solar, the Islamic (Hijri) year is shorter than the Gregorian year by about 11 days, and months in the Islamic (Hijri) year are not related to seasons, which are fundamentally determined by the solar cycle.

Hiragana The Japanese cursive script. Each hiragana character represents a phonetic syllable. See Katakana.

HKL See Input language handle (HKL) and Language/layout pair.

Hub and spoke model Within the context of the .NET Framework, the hub is the main assembly that contains the nonlocalizable executable code and the resources for a single culture, called the "neutral" or "default" culture. The default culture is the fallback culture for the application. Each spoke connects to a satellite assembly that contains the resources for a single culture, but does not contain any code.

Ideographic character A character of Chinese origin representing a word or a syllable that is generally used in more than one Asian language. Sometimes referred to as a "Chinese character." See Kanji, Hanzi, and Hanja.

Indexing Service A base service of Windows NT, Windows 2000, and Windows XP that extracts content from files and constructs an indexed catalog to facilitate efficient and rapid searching. Indexing Service can extract both text and property information from files on the local host and on remote, networked hosts.

Input context An internal structure that stores IME-related status information. Windows supports multiple IME contexts, automatically creating an input context for each active thread.

Input language handle (HKL) A data type to indicate language/layout pairs.

Input locale Pairing of input language (LANGID) and method of input determines what language is currently being entered and how. See Locale, System locale, User locale, and User-interface (UI) language.

Input method Any method used to enter text. These methods include different keyboard layouts and IMEs, as well as newer input services such as voice-recognition engines or handwriting-recognition engines.

Input Method Editor (IME) A program that performs the conversion between keystrokes and ideographs or other characters, usually by user-guided dictionary lookup.

Input Method Manager (IMM) The module on Windows 2000 and Windows XP that handles communication between IMEs and applications.

Internationalization Term used outside of Microsoft to indicate globalization and localizability. I18N is a common abbreviation for "Internationalization" because the "I" in "International" is followed by 18 letters and ends with the letter "N." See Globalization and Localizability.

I18N Abbreviated form of INTERNATIONALIZATION. I + 18 characters + N.

Invariant culture Whereas the neutral culture is associated with a language, but not with any particular country or region, the invariant culture is neither associated with a language nor with a particular country or region. It can be used in almost any method in the Globalization namespace that requires a culture. The invariant culture must be used only by processes that require culture- and language-independent results, such as system services; otherwise, it produces results that might be linguistically incorrect or culturally inappropriate.

ISO International Organization for Standardization. It is a worldwide federation of national-standards bodies.

ISO 4217 currency symbol Three-letter ISO codes for representing currencies and funds (such as CAD for the Canadian dollar).

ISO 8859 The International Organization for Standardization's 8-bit encoding that served as the basis for the Windows (ANSI) code page. Variants of this standard (for example, 8859-2, 8859-5, 8859-13) target different scripts, and each variant corresponds to different Windows code pages.

ISO 10646 The International Organization for Standardization's encoding that is code-for-code equivalent to Unicode.

Isolate, initial, medial, and final character forms The different shapes of an Arabic character that correspond to its position in a word.

Item In scripts, a character string having all the same script and direction attributes.

Jamos The 24 basic elements of the Korean script.

Japanese Emperor Era The Japanese calendar that works exactly like the Gregorian calendar, except that the year and era are different. The Japanese calendar recognizes one era for every emperor's reign. The current era is the Heisei era, which began in the Gregorian calendar year 1989. The era name is typically displayed before the year. For example, the Gregorian calendar year 2002 would be Heisei 14 in the Japanese calendar.

Johab The Korean standard character set (KS C-5601-1992), which corresponds to Windows code page 1361. This character set includes all possible Hangul character combinations.

Kana The set of Japanese hiragana and katakana characters.

Kanji The Japanese name for ideographic characters of Chinese origin. See Ideographic character,Hanzi, and Hanja.

Kashida Character added to justify lines and paragraphs in Arabic. ("Kashida" means "stretch" in Arabic.)

Katakana A Japanese script of phonetic syllables, chiefly used to spell words borrowed from other languages. Each katakana character represents a phonetic syllable. See Hiragana.

Keyboard layout A standard arrangement of characters on a keyboard that defines which keys produce particular characters or scan codes.

Korean Tangun Era calendar According to Korean legend, the god-king Tangun founded the Korean nation in BC 2333. Early Korea used a lunar calendar. As the rest of the world encroached on Korea, it eventually went to the solar, Gregorian calendar. Yet much of the country still uses the lunar calendar to keep track of births and deaths and some traditional holidays.

Korean KSC Character encoding established by the Korean Industrial Standards Association. Now known as "KSX."

KS C-5601-1987 The multibyte Wansung encoding standardized by Korea.

KS C-5601-1992 The multibyte Johab encoding standardized by Korea.

Language enabling (1) Adding support to software for document content in a particular language. In this sense, to enable an application for Japanese means to modify the software so that the user can enter, display, edit, and print text containing Japanese. (2) Modifying software so that it can be localized into a particular language. In this sense, enabling for Japanese means to modify software so that it can display Japanese text correctly in menus, dialog boxes, and other user-interface elements. Note that in either sense, an enabled product can still have the user interface in English, such as when the product is not localized.

Language group Term used to describe the supported script families in Windows 2000.

Language ID (LANGID) A 16-bit value defined by Windows, consisting of a primary language ID and a secondary language ID. Used as a parameter to several Win32 functions and messages.

Language/layout pair (1) A language installed on the system and the input method associated with it. (2) The input language.

Latin script The set of 26 characters (A-Z) inherited from the Roman Empire that, together with later additions, is used to write languages throughout Africa, the Americas, parts of Asia, Europe, and Oceania. The Windows Latin 1 character set covers Western European languages and languages that use the same alphabet, and the Latin 2 character set covers many languages in Central and Eastern Europe. In addition, there are other Windows code pages that support Turkic and Baltic languages written in the Latin script.

Layout The order, positioning, and spacing of text or other user-interface elements.

Lead-byte The first byte of a 2-byte code point in a DBCS code page. See Double-byte character set (DBCS).

Leading characters Characters-such as opening quotation marks, opening parentheses, and currency signs-that shouldn't be separated from succeeding characters.

Left-to-right (LTR) text Text that flows from left to right. There are two primary kinds of text: left-to-right text such as in English and Latin languages, and right-to-left (RTL) text such as in Arabic and Hebrew.

Left-to-right embedding (LRE) mark In a document, it signals that a piece of text is to be treated as embedded left to right. For example, an English quotation in the middle of an Arabic sentence could be marked as being embedded left to right. (LRE affects word order, not character order.)

Left-to-right override (LRO) mark A Unicode control character (U+202D) that forces characters following it to be treated as strong left-to-right characters. Allows for nested directional overrides of bidirectional characters. SeeRight-to-left override (RLO) mark.

Letter (1) The basic element of an alphabet. (2) A higher level of abstraction than Character. For example, both the Spanish "ch" and the Danish "aa" can be considered as single letters for some purposes. (Both sort as a single character.) See Text element and Alphabet.

Levels of localization The amount of translation and customization necessary to create different language editions. The levels, which are determined by balancing risk and return, range from translating nothing to shipping a completely translated product with customized features.

Ligature Two or more characters combined to represent a single typographical character. The modern Latin script uses only a few. Other scripts use many ligatures that depend on font and style. Some languages, such as Arabic, have mandatory ligatures; other languages have characters that were derived from ligatures, such as the German ligature of long and short "s" (ß) and the ampersand (&), which is the contracted form of the Latin word "et."

Linear gradient brush In GDI+, can be horizontal or vertical. For example, a horizontal gradient brush can be configured to change color as you move from the left side of a figure to the right side.

Literal In program code, a string surrounded by double quotation marks or a character surrounded by single quotation marks.

Little-endian A computer architecture that stores multibyte numerical values with the least significant byte values first. On systems using little-endian architecture, the letter "A" (U+0041) is stored as 0x41 0x00. See Big-endian.

Locale The collection of features of the user's environment that is dependent on language, country/region, and cultural conventions. The locale determines conventions such as sort order; keyboard layout; and date, time, number, and currency formats. In Windows, locales usually provide more information about cultural conventions than about languages.

Locale-aware Exhibiting different behavior or returning different data, depending on the locale. For example, the Win32 sorting functions return different results depending on the locale parameter sent to each function.

Locale ID (LCID) A 32-bit value defined by Windows that consists of a language ID, a sort ID, and reserved bits.

Locale-sensitive Exhibiting different behavior or returning different data, depending on the locale. For example, the Win32 sort functions return different results depending on the locale parameter sent to each function.

Localizable resource Any element of a program's UI that requires translation or modification for different languages. These elements are either UI resources or resources that need to be modified for the adaptation of a localized product (font information, locale information, folder names, account names, and so on).

Localizability The design of the software code base and resources such that a program can be localized into different language editions without any changes to the source code.

Localization The process of adapting a program for a specific local market, which includes translating the user interface, resizing dialog boxes, customizing features (if necessary), and testing results to ensure that the program still works.

L10N Abbreviated form of LOCALIZATION. L + 10 characters + N.

Localization kit A subset of tools, source files, and binary files that can be used to create a localized edition of a program. Generally given to translators or third-party contractors.

Logical order In the same order in which it is typed. Generally refers to text that might be displayed in a different order, such as Arabic, Hebrew, or bidirectional text. See Visual order.

Logograph, or logographic From the Greek "logo," meaning "word": a letter, symbol, or sign used to represent an entire word. Chinese characters are more properly termed "logographic" than "ideographic" because they represent words or parts of words rather than abstract concepts.

Lowercase Denotes letters that are not capitalized. For instance, the word "nationality" is all lowercase. The notion of lowercase does not apply to East Asian and Middle Eastern scripts.

Mark In typography, a glyph for a character like a diacritic or tone mark that combines with other marks or characters. Marks can be spacing or nonspacing glyphs in a font.

Mark attachment Used in typography when marks need to be attached or positioned to other marks, base glyphs, or ligatures.

Message table A Win32 resource that uses sequential numbers rather than escape letters to mark replacement parameters, making it convenient to store alert messages and error messages that contain several replacement parameters.

Microsoft Layer for Unicode (MSLU) Enables Unicode applications to run on code page-based versions of Windows (Windows 95, Windows 98, and Windows Me).

Mirroring System-provided support that offers a true right-to-left (RTL) look and feel to the user interface when creating localized applications for RTL languages (such as for Arabic and Hebrew versions of Windows 98, Windows Me, Windows 2000, and Windows XP).

Mixed environment A computer environment, usually a network, in which the operating systems of different computers are based on different character encodings.

MLang A Component Object Model (COM) component that provides a variety of services. These services include detecting the character encoding used by Web pages and e-mails, converting text from one encoding to another as part of an import or export operation, and displaying characters that are not included within the font specified for parts of a Web page.

MM_TEXT Unit of measure used to convert logical units to device units; it also defines the orientation of the device's x - and y -axes. GDI uses the mapping mode to convert logical coordinates into the appropriate device coordinates. The MM_TEXT mode allows applications to work in device pixels, where 1 unit is equal to 1 pixel. The physical size of a pixel varies from device to device.

Mode biasing Incorporation of logic that biases the method used for input (mode) toward the type of input that is expected. For example, the name field on a Contacts dialog box will have information telling the input method that people's names are expected. With this information, the input method biases toward those results and thus provides more accurate input.

Modeless input A method of entering East Asian-language input with an IME. In contrast with modal input, modeless input allows you to easily and seamlessly switch back and forth between the composition mode and the direct mode (the document itself). You can easily correct, navigate, and make input to the general document, whereas with modal input the rest of the document is temporarily unavailable as long as you are in composition mode.

Morpheme The smallest meaningful unit of a word. The word "dog" is one morpheme. The word "dogs" is two morphemes: "dog" + the plural marker "s." Many ideographs are based on morphemes.

Multibyte character set (MBCS) A character encoding in which the code points can be either 1, 2, or more bytes. See Double-byte character set(DBCS) and CJK/CJKV.

Multilingual Supporting more than one language simultaneously. Often implies the ability to handle more than one script or character set.

Multilingual User Interface (MUI) Pack A set of language-specific resources that can be added to the English version of Windows XP Professional and the .NET Server. Once installed, MUI Packs allow the UI language of the operating system to be changed to one of 33 supported languages, depending on user preference. The MUI Pack is the same as the MultiLanguage version of Windows 2000 Professional and Server, though it provides additional functionality.

Multiple-document interface (MDI) A UI in an application that allows the user to have more than one document open at the same time.

Mutex A synchronization object that ensures only one thread at a time can access a shared resource. A thread must have ownership of the mutex before it can access the resource. The thread becomes blocked if the mutex is owned by another thread.

National standard A linguistic rule, measurement, educational guideline, or technology-related convention as defined by a government or by the International Organization for Standardization. Examples include character sets, keyboard layouts, and some cultural conventions, such as punctuation.

Network News Transfer Protocol (NNTP) The Internet protocol that governs the transmission of newsgroups.

Neutral character A character whose directionality (right-to-left or left-to-right) is dependent on the directionality of the characters that surround it. See Contextual analysis.

Neutral culture In the .NET Framework, refers to cultures identified by language only (with no associated geographic region). A neutral culture is indicated by a two-letter code, such as "de" for German.

NLS API Acronym for National Language Support API. The set of system functions in 32-bit Windows containing information that is based on language and cultural conventions.

No-compile localization A process of localization where the code is not recompiled.

Noncompile mirroring Activating mirroring for the binary dynamic-link library resource files of an application, without the need for rebuilding the application.

Nonspacing character A character, such as a diacritic, that has no meaning by itself but overlaps an adjacent character to form a third character.

OpenType An extension of the TrueType font format, adding support for PostScript font data.

OpenType Layout An extension to the OpenType format designed to provide support for international and high-end typography.

Original Equipment Manufacturer (OEM) See OEMCP.

OEMCP Default OEM code page of the system. The OEM code page is used for conversions of MS-DOS-based, text-mode applications.

Outline A series of contours made up of straight lines and curves that define the shape of a glyph.

Overflow characters Punctuation characters that are allowed to extend beyond the right margin for horizontal text or below the bottom margin for vertical text.

Overload Within the context of the .NET Framework, the concept of defining a procedure in multiple versions, using the same name but different argument lists. The purpose of an overload is to define several closely related versions of a procedure without having to differentiate them by name. You do this by varying the argument list.

Path gradient brush In GDI+, a path gradient brush can be configured to change color as you move from the center of a figure toward the boundary.

Phoneme A unique individual sound used in a language.

Pilot language A language that is localized before other languages; this language is used to define the localization process and to test for localizability issues. See localizability, localization.

Plaintext Computer-encoded text that contains only code elements and no other formatting or structural information (for example, font size, font type, or other layout information). Plaintext exchange is commonly used between computer systems that might have no other way to exchange information.

Pop directional formatting (PDF) In HTML, PDF terminates the effects of the last explicit code (either embedding or override) and restores the bidirectional state to what it was before the last left-to-right embedding (LRE), right-to-left embedding (RLE), right-to-left override (RLO), or left-to-right override (LRO) control characters.

Positional forms Refers to the shape of a character that varies with the character's position in a word. For example, the Arabic character "ha" can take any of four shapes, depending on whether the character stands alone or whether it falls at the beginning, middle, or end of a word.

Precomposed character A character that is equivalent to a sequence of one or more characters. It is also known as a "composed character" or a "composite character." Thus the combining character sequence "a + h" forms the precomposed, composed, or composite character "à."

Private-use zone The area in Unicode repertoire from U+E000 through U+F8FF, U+F0000 through U+FFFFD, and U+100000 through U+10FFFD that is set aside for vendor-specific or user-designed characters.

Radicals A group of strokes in a Chinese character that are treated as a unit for the purposes of sorting, indexing, and classification. A character can contain more than one element that is recognized as a radical, but each character contains only one element, called the "main radical," that is used as the indexing radical. The main radical often gives a hint as to the general meaning of the character, and other radicals in the character might indicate how the character is pronounced.

Rapid Application Development Provides the ability to develop and deploy applications quickly by automating much of the development process and eliminating repetitive tasks.

RCDATA resource A custom Windows resource element.

Reading order The overall direction of a sequence of text. Whereas words in a given script always flow in the direction associated with that script (for example, LTR for Latin, RTL for Arabic and Hebrew), the flow of the sentence itself depends on the reading order. For example, a mixture of Arabic and French text can be regarded as French-embedded in an overall Arabic sentence, implying RTL reading order, or as Arabic-embedded in French, implying LTR reading order.

Registry A Windows file that stores user preferences, including international settings as well as application-specific settings.

Release delta The time between the release of the domestic product and the release of the localized edition.

Remoting In the .NET Framework, remoting allows objects to interact with one another across application domains. The framework provides a number of services, including activation and lifetime support, as well as communication channels responsible for transporting messages to and from remote applications.

Rendering The way in which a character or a character sequence is graphically displayed. See Complex scripts, Contextual analysis, and Uniscribe.

Request For Comments (RFC) documents The written definitions of Internet protocols and policies.

Resource (1) An element, such as a string, icon, bitmap, cursor, dialog, accelerator, or menu, that is included in a Windows resource (.RC) file. (2) Any item that needs to be translated.

Rich text Text saved with formatting instructions that multiple applications, including compatible Microsoft applications, can read and interpret.

Right-to-left (RTL) text Text flows from right to left. Examples are Arabic and Hebrew.

Right-to-left embedding (RLE) mark In a document, it signals that a piece of text is to be treated as embedded right to left. For example, a Hebrew phrase in the middle of an English quotation could be marked as being embedded right to left. (RLE affects word order, not character order.)

Right-to-left override (RLO) mark A Unicode control character (U+202E) that forces characters following it to be treated as strong right-to-left characters. Allows for nested directional overrides of bidirectional characters. See Left-to-right override (LRO) mark.

Romaji A writing system based on the Latin alphabet that is used to represent Japanese text.

Round-trip conversion Mapping a character from one character encoding to another and back. Of particular interest is how well information is preserved during round-trip conversion.

Ruby An annotation or pronunciation guide for a string of text. The string of text annotated with ruby text is referred to as the "base text."

Run-time library Functions included with a C compiler that programs can call to perform various basic operations.

Runs In scripts, portions of an item that have continuous formatting attributes.

Scan code The value sent from the keyboard to the keyboard driver that represents which key was pressed.

Screen coordinates The system and applications specify the position of a window on the screen in screen coordinates. The full position of a window is often described by a RECT structure containing the screen coordinates of two points that define the upper-left and lower-right corners of the window.

Screen dump A bitmap of an element in a program's graphical user interface, such as a dialog box or menu.

Script A collection of characters for displaying written text, all of which have a common characteristic that justifies their consideration as a distinct set. One script can be used for several different languages (for example, the Latin script, which covers all of Western Europe). Some written languages require multiple scripts (for example, Japanese, which requires at least three scripts-the hiragana and katakana syllabaries and the kanji ideographs imported from China). This sense of the word "script" has nothing to do with programming scripts such as Perl or Visual Basic Scripting Edition (VBScript).

Scripting engine Component that handles character line measurement, display, caret movement, character selection, justification, shaping, and line breaking for complex scripts.

Separators Symbols used to separate items in a list, mark the thousands place in numbers, or represent the decimal point. Different locales follow different conventions for separators.

Setup project Allows you to create installers in order to distribute an application in Visual Studio .NET.

Shift-JIS The multibyte encoding developed by Microsoft for Japanese that is based on Japan Industry Standard (JIS) standard X 0208. The name comes from the way the lead bytes in Shift-JIS shift around the encoding range of half-width katakana in JIS X 0208.

Shortcut key A keyboard combination that activates a program command directly, as an alternative to activating the command through the program menus.

Simple Mail Transfer Protocol (SMTP) A protocol for sending messages from one computer to another on a network; used on the Internet to route e-mail.

Simplified Chinese The Chinese script used in the People's Republic of China and Singapore. It consists of several thousand ideographic characters that are simplified versions of traditional Chinese characters.

Simultaneous ship, or "sim ship" The release of localized editions of a product at the same time as the domestic product; or a short release delta, usually within 30 days (to allow for nondevelopment needs).

Single binary A functional binary that is fully globalized and can be used as is for any language version of the software.

Single-byte character set (SBCS) A character encoding in which each character is represented by 1 byte. Single-byte character sets are mathematically limited to 256 characters.

Slant The obliqueness or tilt of the glyphs in a font. The most common slants are "regular" and "italic."

Smart tags A technology introduced in Office XP that provide users with the ability to associate text and data with actions. Smart tags can provide actions for names, dates, times, telephone numbers, addresses, stock ticker symbols, and so on.

Software Development Kit (SDK) A set of tools and libraries for creating software applications for Windows operating systems.

Sort keys Numeric representations of a sort element based on locale-specific sorting rules. A sort key consists of several weighted components that represent a character's script, diacritics, case, and so on.

Spacingcharacter A character with a nonzero width.

Specialized word break and justification Refers to scripts, such as Thai, that have complex rules for dividing words between lines or justifying text on a line.

Specification, or "spec" A detailed plan of a program's user-interface design and the expected functionality of program features.

SQL-92 Designed by what was formerly the National Committee for Information Technology Standards (NCITS)-now known as the "InterNational Committee for Information Technology Standards" (INCITS)-Technical Committee H2 on Database to be a standard for relational database management systems (RDBMSs).

SQL Query Analyzer A graphical user interface for designing and testing Transact-SQL statements, batches, and scripts interactively.

SQL Server Enterprise Manager Allows enterprise-wide configuration and management of SQL Server and SQL Server objects.

Status window The window of an IME in which the user can change the IME's conversion mode or input mode.

Stroke count The number of strokes it takes to draw an ideographic character.

Strong character A character from which text direction can be determined.

Substitutions English, French, and other languages based on Latin can substitute a single ligature, such as "5," for this particular ligature's component glyphs "f" and "i." Conversely, the individual "f" and "i" glyphs could replace the ligature, possibly to give a text-processing application more flexibility when spacing glyphs to fill a line of justified text.

Syllabary A set of written characters in which each character represents a syllable (for example, a consonant sound followed by a vowel sound). Examples of syllabaries include Japanese katakana and hiragana and the Indic scripts.

System locale The system locale determines the Windows code page used by the ANSI (non-Unicode) version of Win32 APIs. String and character parameters passed to a Win32 ANSI are converted from this Windows code page to Unicode.

Taiwan calendar The Taiwan calendar works exactly like the Gregorian calendar, except the year is different. Years in Taiwan are calculated from the year 1912, when Dr. Sun Yat-sen founded a republic in China. 1912 represents Year 1 of the Taiwan calendar. To calculate the year in the Taiwan calendar, take the year in the Gregorian calendar and subtract 1911. For instance, the Gregorian calendar year 1971 would be 60 in the Taiwan calendar.

Text element A script's smallest unit of text that can be displayed or edited.

Text Object Model (TOM) interfaces A substantial set of text-manipulation interfaces. Text solutions such as Microsoft Word and rich edit controls support the TOM feature set. Since rich edit controls ship with Windows operating systems, they are the standard means of obtaining TOM functionality.

Thai calendar The Thai calendar uses the Buddhist Era (BE), which is 543 years older than the Christian Era (AD). To convert from a BE date to an AD date, subtract 543. Thus BE 2543=2000. Before April 1, 1889 AD, Thailand used a lunar calendar of 12 or 13 months each with 29 or 30 days, each month starting with the new moon. The Gregorian calendar was adopted on April 1, 1889.

Thread locale The locale of a given thread. Gets inherited upon creation from the current user locale and can be changed at run time to any valid locale (per thread). Calls to NLS APIs can use this locale to format numbers, date, and time.

Traditional Chinese The set of Chinese characters, used in such countries/regions as Hong Kong SAR, Macau SAR, and Taiwan, that is consistent with the original form of Chinese ideographic characters that are several thousand years old.

Trail-byte The second byte of a 2-byte code point in a DBCS code page. See Double-byte character set (DBCS).

Transact-SQL A method to communicate with and access data on SQL Server. Applications that communicate with SQL Server do so by sending Transact-SQL statements to the server, regardless of an application's UI.

TrueType A digital font technology (designed by Apple Computer and now used by both Apple and Microsoft in their operating systems) that offers superior display quality on computer screens and printers. See Bitmap font.

Typeface Name given to a particular style of text. In contrast, a font is an implementation of a typeface.

Typography The process of displaying text in a variety of fonts, sizes, and styles.

Umlaut The two dots placed above a vowel, such as "ä," "ö," and "ü," which are used in German and other European languages to indicate a change in the pronunciation of the vowel. See Diaeresis.

Unicode A worldwide character encoding that includes most of the world's scripts; it is developed, maintained, and promoted by the Unicode Consortium, a nonprofit computer industry organization. (The official Unicode Consortium Web site is http://www.unicode.org.)

Uniscribe The shaping and layout engine for complex scripts used by the latest versions of Windows, Office, and Internet Explorer. See Complex scripts, Contextual analysis, and Rendering.

Uppercase Denotes letters that are capitalized. For instance, acronyms (such as "HTML") typically consist of all uppercase letters. The notion of uppercase does not apply to East Asian and Middle Eastern scripts.

Usability testing A series of tests in which users are observed trying to complete a given set of tasks. The purpose of usability testing is to determine how intuitive test subjects find new program features.

User assistance (UA) Refers to any form of documentation (including printed documents, online documents, and Help files) that corresponds to a particular product.

User-defined character See End-User Defined Character (EUDC).

User-interface (UI) language Language in which the operating system displays its menus, Help files, and dialog boxes. See Locale, System locale, User locale, and Input locale.

User locale The user preferences for formatting of dates, currencies, numbers, and so on. The user locale is a per-user setting, and does not require the user to restart or to log off or log on the computer.

UTC Coordinated Universal Time (often spelled out as "Universal Coordinated Time"). It is the standard time common to every place in the world. Formerly and still widely called "Greenwich Mean Time" (GMT) and also "World Time." It is expressed using a 24-hour clock.

Version stamp In Windows, the information included in the resource file that specifies the company name, application name, copyright, version number, and language edition of a program.

Vertical kerning A feature used in Arabic to position diacritics at different heights, since it is considered rudimentary to display diacritics all at the same height.

Vertical metrics A collective term to indicate information from a set of fields that is available in various font tables for determining vertical spacing when laying out text. Some of these fields can be used to arrive at the vertical distance required between two consecutive horizontal baselines, while others can be used to determine vertical spacing for each glyph when text is laid out vertically.

Visual C++ Microsoft's object-oriented C compiler.

Visual order The ordering used to display glyphs on a screen, printed page, or other medium. Usually used with bidirectional text, because reordering is required to go from logical order to visual order. See Logical order.

Wansung The Korean standard character set (KS C-5601-1987), which corresponds to Windows code page 949. It covers the most common Hangul character combinations. Extended Wansung covers all possible Hangul combinations.

-W APIs The Win32 API entry points that expect string parameters to be wide characters (encoded in Unicode). See -A APIs.

wchar_t The ANSI C-defined wide-character type, usually either 16 or 32 bits. ANSI rules say that wchar_t should be at least as wide as the char data type, and that the wide-character equivalents of the C language source character set should be created by simple zero or sign extension.

Web Forms Generally applies to the design-time technology that enables the authoring of ASP.NET pages in a visual designer.

Web Parts Reusable components that contain Web-based content, such as XML code, HTML pages, or script. Web Parts support a set of standard properties that determine how the Web Parts are rendered in a Digital Dashboard. Office XP Developer makes it possible for you to create and customize Digital Dashboards and Web Parts. See Digital Dashboard.

Weight The thickness or darkness of glyphs in a font. The most common weights are "regular" and "bold," but some font families can include such weights as "light," "demi," "heavy," "extra bold," and so on.

Wide character A character encoded by a wchar_t or a 16-bit (WORD) data type. Often used to refer to UTF-16-encoded characters.

Win32 API The set of 32-bit functions supported by Windows.

Win32s API A subset of the Win32 API that makes it possible to create a single binary that runs on Windows 3.1 and on all 32-bit versions of Windows.

Windows Forms The new platform for Windows application development, based on the .NET Framework. Provides a clear, object-oriented, extensible set of classes that enables you to develop rich Windows applications. Additionally, Windows Forms can act as the local UI in a multitier, distributed solution.

Windows Forms controls Reusable components that encapsulate UI functionality and are used in client-side Windows applications. Not only do Windows Forms provide many ready-to-use controls, they also provide the infrastructure for developing your own controls.

Windows Forms Designer Provides a rapid development solution for creating Windows applications. It is the locus of visual, client-based forms design. Using the designer, you can add components, data controls, or Windows-based controls to a form.

Windows Services The idea (originating with MS-DOS) of a program operating in the background while a user is doing something else in the foreground has evolved on Windows systems to a Windows Service. Examples include plug-and-play device detection, running message queues, file indexing, and task scheduling. In addition to the services the operating system installs itself, programs such as SQL Server install their own Windows Services to implement functionality that must be available to all users.

World-ready A program that has been properly globalized and developed for ease of localization (known as "localizability"). See Localizability.

Writing system The collection of scripts and orthography required to represent a given human language in visual media.

XML Acronym for Extensible Markup Language. An open standard for exchanging structured documents and data over the Internet that was introduced by the World Wide Web Consortium (W3C) in November 1996. XML is a simplified version of Standard Generalized Markup Language (SGML).


Organizing a Product Team

Footnotes

Chapter 1

1. JPN=Japanese; KOR=Korean; CHS=Simplified Chinese; CHT=Traditional Chinese; Bi-Di=bidirectional languages such as Arabic and Hebrew.

Chapter 3

1. At the time of this writing, the most up-to-date version of the standard is Unicode 3.2, available on the Unicode Consortium Web site at http://www.unicode.org.

Chapter 4

1. OEM code pages are used in Microsoft MS-DOS; most of them, including code page 852, were created by IBM.

2. "Unicode sort weight" is a legacy term used to describe the combination of script membership and alphabetic weights in collation. This term does not imply conformance to the Unicode Collation Algorithm.

3. The U.S. Customary System is a derivative of the British Imperial System. There are some differences between the two systems. For instance, a U.S. gallon is not the same as an Imperial gallon. In a similar vein, a U.S. billion is 1 thousand million, whereas a British billion is 1 million million (which in the United States would be a trillion).

Chapter 6

1. For simplicity, this chapter will generally use the shortened product name of Windows 2000 (rather than Microsoft Windows 2000 Professional), and of Windows XP (rather than Microsoft Windows XP Professional and Microsoft Windows XP Embedded).

2. The MUI Pack cannot be installed on localized versions of Windows 2000 nor Windows XP.

Chapter 13

1. You need to make sure that the corresponding scripts are enabled on your particular computer. (You can do this through the Regional And Language Options property sheet in Windows XP.)

2. The East Asian scripts fall back to Microsoft Sans Serif, which is the font used for font linking. (For more information on font linking, see Chapter 5). Microsoft Sans Serif is linked to the East Asian font that matches the system locale of the machine. The fonts are Gulim for Korean, MingLiu for Traditional Chinese, MS Gothic for Japanese, and SimSun for Simplified Chinese.

Chapter 18

1. Windows XP implies the Microsoft Windows XP Home Edition, Microsoft Windows XP Professional, and Microsoft Windows .NET Server.

2. The Windows code-page versions of all APIs are required in order to allow existing code page-based applications to run on Windows NT/2000/XP without any modifications. When it was decided in 1991 that the Windows NT platform would be purely Unicode-based, Unicode was still not well known, and thus code-page support was imperative.)

3. For example, locale information that is available only in Unicode on Windows NT (such as for Indic locales) is not available on Windows 95/98/Me.