Hungarian  LATIN2                      Turkish        LATIN5,3
    Icelandic  LATIN1                      Ukranian       CYRILLIC

          Table 3: Preferred Transfer Syntax Character Sets

_____________________________________________________________________________


IMPLEMENTATION OF LEVEL 1

The Level-1 Kermit extension can be added to existing Kermit programs with
a minimum of effort.  The following steps are required for each Kermit program:

1. Add the SET FILE TYPE { BINARY, TEXT } command, if the program doesn't
   have it already.  SET FILE TYPE TEXT enables text-file character set
   conversion at all levels.  SET FILE TYPE BINARY disables conversions of
   all kinds.

2. Add the SET FILE CHARACTER-SET <name> command.  The set of <names> should
   include ASCII (used for program source, etc) plus the names of any
   "national" character sets that are used on this particular computer.

3. Add the SET TRANSFER-SYNTAX <option> command.  The options should include
   NORMAL (Kermit's normal, unextended ASCII-based transfer syntax), as well
   as CHARACTER-SET <name>, including the names of one or more character sets
   from Table 2 which contain the characters from the computer's local
   character set(s) in (1). 

4. Add translation tables between each pair of character sets in (2) and (3).
   For each pair, two translation tables are necessary: one from the local
   set to the common one, and one from the common set to the local one.

5. Add SHOW commands to let the user find out (a) what character sets are
   available, and which ones are currently selected, for the transfer syntax
   and for local files.

6. Optionally, to allow for user-defined character-set translations, also
   add the LOAD TRANSLATION-TABLE, SHOW TRANSLATION-TABLE, and DUMP
   TRANSLATION-TABLE commands (described in the next section).

Example: To transfer a Finnish-language text file from a computer that uses
the Finnish ISO 646 national replacement set to an IBM PS/2, and to store the
file using the PS/2's Multilingual Code Page:

  On the sending computer:                   On the receiving computer:
    SET FILE TYPE TEXT                         SET FILE TYPE TEXT
    SET FILE CHARACTER-SET FINNISH-ISO646      SET TRANSFER-SYNTAX LATIN1
    SET TRANSFER-SYNTAX LATIN1                 SET FILE CHARACTER-SET CP850

To transfer a C-language source program between the same two computers:

  On the sending computer:                   On the receiving computer:
    SET FILE TYPE TEXT                         SET FILE TYPE TEXT
    SET FILE CHARACTER-SET ASCII               SET TRANSFER-SYNTAX LATIN1
    SET TRANSFER-SYNTAX LATIN1                 SET FILE CHARACTER-SET ASCII


TRANSLATION TABLES

In some cases, translation tables will be 1-for-1.  That is, the two character
sets are the same size, and each character from one set can be found in the
other set.  In such cases, the translation table need be only a list of
numbers, in which position "n" in the table contains the translation for
character number "n".

In many cases, the two character sets will be the same size, but certain
characters from one will be lacking in the other, and/or vice versa.  For
example, IBM Code Page 850 and the Apple Macintosh sets are both "8-bit
ASCII", but the IBM set lacks the Macintosh's Y-umlaut, and the Macintosh
set lacks IBM's Y-acute.

In other cases the character sets will be different sizes.  We have long been
familiar with this problem when translating between 7-bit ASCII and 8-bit
EBCDIC.  In Japan, there must be translations between single-byte Roman,
Greek, and Cyrillic characters and the two-byte JIS X 0208 character set.  In
the future we must allow for the possibility of a multibyte universal
character set containing characters from many languages, as proposed in ISO
DIS 10646.  Here, the translation will be between 7- or 8-bit one-byte codes
and a multibyte code.

It is recommended that translation tables built into Kermit programs be as
general and useful as possible, substituting the closest possible character
when an exact match is not available.  For instance, when translating from
Latin-1 to ASCII, accented letters should be translated into the corresponding
unaccented letters: a-acute becomes a, etc.

Translation tables should be designed for maximum flexibility.  They should
not be restricted 1-to-1, byte-for-byte mappings, but should also allow for
many-to-single and single-to-many byte mappings.  Translation tables are best
left to native speakers of each language.

USER-DEFINED TRANSLATIONS

It should be possible for users to alter Kermit's translation tables or to add
new ones, without having to change the program's source code.  For example, in
certain situations it might be preferable to have a-grave rendered in ASCII as
"a", but in others as "`a", "a`", or even "?".  It is also likely that new
character sets will appear which are unknown to the Kermit program.

To return to the example of translating Latin-1 into ASCII, consider German
text containing the Ess-Zet character and vowels with umlauts.  It is
acceptable to render Ess-Zet as "ss", and to render an umlaut vowel as the
same vowel followed by "e".  But this should not necessarily be done for
languages other than German.  Therefore German users might wish to set up a
special translation table which performs these functions, so that
"Gr<u-umlaut><ess-zet> aus K<o-umlaut>ln" would become "Gruess aus Koeln"
(correct German) rather than "Grus aus Koln".

For these reasons, we suggest a standard format for translation tables, and a
LOAD TRANSLATION-TABLE command to allow the user to add new character sets to
a Kermit program's repertoire, or to alter current translations.

Each table within a program is assigned an arbitrary tablename.  For example,
LATIN1-CP850 could be the name for the Latin-1 to CP850 table in the PS/2, and
CP850-LATIN1 could be the name for the table in the other direction.  To load
a replacement table, the user would issue the command:

  LOAD TRANSLATION-TABLE <tablename> <filename>

where <tablename> is the name to assign to the new table.  If a table with
that name already existed, that table is replaced.  The layout of a loadable
translation table is given in Appendix C.

A Kermit program, upon loading one of these files, would set up the
translation table, add the name of the table and of the character sets
themselves to the appropriate keyword tables, and so on.

So that the translation-table related commands can also be effective for
built-in translation tables, it is recommended that the built-in tables be
designed in the same format as the loadable tables.

Two additional commands should be furnished to allow the user to get
information about the currently loaded tables:

  SHOW TRANSLATION-TABLE <tablename>

which would give summary information, and:

  DUMP TRANSLATION TABLE <tablename> <filename>

which would write out a translation table (even a built-in one) in the form
shown in Appendix C, so that it could be edited and loaded again.


ATTRIBUTE PACKETS AT LEVEL 1

The objective of Kermit's Level-1 extension is to accommodate as many
computers as possible with a minimum of programming effort.  But this approach
places a burden on the user in the form of new commands and the confusion
which results if the user forgets to issue these commands.

Level 1 does not require support for Kermit File Attribute Packets, whose use
is negotiated in the Kermit Initialization exchange.  But the user's burden
can be alleviated if the sending Kermit program uses an attribute field to
inform the receiving Kermit of the character set to be used in the transfer
syntax.  The receiving program can accept or refuse the file based on whether
it supports the specified character set.  If the receiving program refuses a
file, the user can override this refusal, for example, if a long file contains
only a word or two in an unknown character set.  The most common user-override
is the command SET ATTRIBUTES OFF.  However, this also disables other
desirable effects of attribute packets, such as prenotification of file size.
Therefore, it might be desirable to let the user specify exactly which
attributes are to be "turned off", e.g. SET ATTRIBUTES CHARACTER-SET OFF.

In order for the sender to inform the receiver of the transfer alphabet, a new
value for the Encoding attribute ("*") is defined, namely "C", which is
substituted for the normal value "A" (ASCII).  "C" means that the actual
character set is specified in a separate new character-set attribute, "2".
The operand of the character-set attribute is the letter "I" (for ISO)
followed by an ISO registration number for the character set, as shown in
Table 2, expressed in decimal ASCII digits, for example:

  +---+---+---+  +---+---+------+
  | * | ! | C |  | 2 | $ | I100 |
  +---+---+---+  +---+---+------+

where "*" is code for the Encoding Attribute (or transfer syntax), "!" is the
length (ASCII 33 - 32 = 1) of its value, and the single character "C" is the
value itself, which means "I'm using a specified Character set, look in the
'2' attribute to find out which one."  If "C" is specified as the encoding,
but the "2" attribute is absent, this should be diagnosed as a protocol error.

"2" is the character set attribute, "$" is the length of the following data
(in this case, ASCII 36 - 32 = 4), and the four characters "I100" mean
"character set ISO 100", i.e. ISO Latin-1.  The "I" is included to allow for
the possibility of other character set registration authorities (for example,
K for Kermit Development and Distribution).

Based on this information, the receiver may accept or reject the file, using
Kermit's normal attribute response mechanism.  To accept, it puts a "Y" as the
first character of the data field, followed by "*" to indicate that it accepts
Level-1 transfer syntax, and a "2" to indicate it accepts the specified
character set.  To refuse, it puts an "N" instead of a "Y", followed by "*" if
it cannot do Level-1, and/or a "2" if it does not recognize the character set.

If the file is refused in this manner, the sending Kermit can issue an
informative message to the user, and the user can cancel the transfer manually
and then find some other way to transfer the file, for example in normal,
Level-0 text or binary mode with pre- and postprocessing, or even by loading a
new translation table.

It is recognized that there are presently Kermit implementations in the USSR,
Japan, and elsewhere that use character sets other than the ones listed in
Table 2 in their transfer syntax, and/or sets that are not listed in the
International Register.  It is recommended that these Kermit programs be
converted to use the recommended standard character sets.

LEVEL 1 PERFORMANCE

Level 1 can be used to transfer files containing special characters when
character-set switching is not required.  However, Level-1 transfer will not
always be efficient.  Since the special characters have their 8th bits set to
one, there will be a lot of 8th-bit prefixing in the 7-bit environment -- the
higher the proportion of special characters to ASCII characters, the lower the
efficiency.  For a language like Russian, in which all letters come from the
right half of the character set, efficiency will be very poor.

Therefore, even though Russian (and Greek, Hebrew, and Arabic) are served by
Level 1 of this proposal, files encoded in these character sets can be
transferred more efficiently using the facilities of Level 2 in the 7-bit
communication environment.  See Table 4.

EXPANDED SYNTAX, LEVEL 2: MULTIPLE CHARACTER SETS

Suppose there is a computer that can store a file containing characters from
many languages.  It may do this by using a multibyte character code, or by
imbedding some kind of control information in the file to mark each change of
character set.

One such computer is the Xerox Star and its successors, described by Joseph D.
Becker in the Scientific American articles "Multilingual Word Processing"
(July 1984) and "Arabic Word Processing" (July 1987).  The Star stores textual
data intermixed with special codes.  A byte of all 1's means "alphabet shift",
followed by another byte or two to identify the alphabet.

Another, more limited, example is a computer using one of the AT&T Extended
UNIX Codes (EUC), such as JAE for Japan.  In this code, a byte with its high
order bit set to zero is ASCII.  If it is set to one, then it is either a
1-byte Kanji or 1-byte Katakana character, or (if it has a certain special
value) a shift character indicating that the next two bytes are a Kanji
character.  (See N. Takahashi and W. Krone, "The Language Problem", UNIX
Review, February 1987.)

A third example is an IBM PC or PS/2 running a commercial word processor which
uses the PC's graphics adapter to display characters from different alphabets
(Roman, Greek, etc) in different renditions (bold, italic, underlined).  A
multilanguage word processor file may contain not only alphabet information,
but also formatting and rendition information.  The format of these word
processor files is proprietary, and differs from product to product.

A final example is a simple IBM PC "8-bit ASCII" text file which also contains
the PC's line-drawing characters.  These characters have no equivalents in the
ISO Latin alphabets, and so two standard character sets would be required to
represent these files during transmission.

Now suppose we want to transfer a multi-language text file from one computer
to a different kind of computer.  Since there will be a growing need to do
this, and a growing number of computers and applications that will support
multi-language text in incompatible ways, it is clearly impractical to require
each computer to know the formats and codes of each other computer.

Once again, a standard common intermediate representation, or transfer syntax,
is required so that each Kermit program need only know the codes and formats
used on its own computer, plus the transfer syntax.  But unlike Kermit's
normal transfer syntax, and unlike Kermit's Level-1 extended transfer syntax,
the multi-language syntax must embody an in-band mechanism for identifying
character sets and switching among them.

Fortunately, these mechanisms are already well-defined in the host-terminal
communications environment, and they can be readily adapted to Kermit file
transfer.  The mechanisms we propose to use are defined in the following
international standards:

  ISO 4873, "Information Processing - ISO 8-bit code for information
    interchange - Structure and rules for implementation"

  ISO 2022, "Information Processing - ISO 7-bit and 8-bit coded character 
    sets - Code extension techniques"

  ISO 2375, "Data Processing - Procedure for registration of Escape Sequences"

  ISO "Internation Register of coded Character Sets to be Used with Escape
    Sequences"

These standards are summarized in Appendix B, "How the Standards Work".


KERMIT MULTI-CHARACTER-SET FILE TRANSFER

Level 2 Kermit syntax is intended for transferring multilanguage files that
cannot be adequately represented in a single standard character set.  The new
"international" transfer syntax preannounces character sets by their ISO
registration numbers, designates them by their registered escape sequences,
and invokes them by single or locking shifts as defined in ISO Standard 2022.

ENABLING LEVEL-2 TRANSFER SYNTAX

The new transfer syntax must be selected in some way, either automatically or
explicitly by the user.  In the automatic case, the Kermit program recognizes
(somehow) that it is to transfer a multi-alphabet text file.  In the manual
case, the user issues the command SET TRANSFER-SYNTAX INTERNATIONAL.

It must also be possible for the user to override the automatic use of
international syntax via the command SET TRANSFER-SYNTAX NORMAL.

PROTOCOL PRIOR TO DATA TRANSFER

It is strongly recommended that any Kermit program which is to use
international syntax also support file attribute (A) packets.  These are used
for two purposes: (1) to inform the receiver that international syntax will be
used and with which ISO-2022 facilities, and (2) to preannounce the file's
character sets.  This will give the receiver the opportunity to refuse files
that it cannot translate, and to allocate the necessary resources for those
files which it can accept.

In Level 2, the value of the encoding attribute "*" should be the uppercase
letter "I" (for International), optionally followed by one or more ISO-2022
announcer letters, as listed in Appendix C, for example "IBZ" to declare that
G0 and G1 will be used with locking shifts, and G2 with single shifts.

  +---+---+-----+
  | * | # | IBZ |
  +---+---+-----+

In addition, the sender may (but is not required to) preannounce the transfer
syntax character sets by listing them in the new attribute, "2", "Character
Sets".  The value of the character-sets attribute is a comma-separated list of
ISO character set registration numbers.  For example:

  +---+---+----------------+
  | 2 | . | 1I00,I127,I144 |
  +---+---+----------------+

where "2" is the character-set attribute, "." is the length of the following
value (in this case "." = ASCII 46 - 32 = 14) and the next 14 bytes list ISO
character sets numbers 100, 127, and 144, each number prefixed by "I" to
denote the ISO registration authority.

If the sending Kermit can ascertain a file's character sets easily, it should
send this information in the attribute packet.  Otherwise preannouncement of
character sets could require a time-consuming scan through the file prior to
sending it, which is undesirable for large files not only because it reduces
Kermit's efficiency, but also because it could cause the entire Kermit session
to time out.  Therefore, preannouncement of character sets is not required.

The receiver may accept or refuse a file using Kermit's normal attribute reply
mechanism.  When accepting the file, it should include, at minimum, the "*"
attribute its acceptance, so that the sender will know that the receiver
understands international syntax.  When refusing a file, it should indicate
what caused the problem: "*" means it can't do international transfer syntax,
but "2" (without "*") means that one or more of the announced character sets
are unknown.

If the file is refused in this manner, the sending Kermit can issue an
informative message to the user, and the user can cancel the transfer manually
and then find some other way to transfer the file (for example, binary mode,
or normal text mode with pre- and postprocessing, or even by loading a new
translation table).

DATA TRANSFER PROTOCOL

Transfer of a multi-character-set text file in international transfer syntax
by Kermit is similar to transfer of a 7-bit ASCII text file, except that it
may contain embedded control characters and escape sequences to identify and
switch between character sets.  The file sender translates the file's
characters (if necessary) into one or more registered alphabets, imbedding
character-set designation and shifting codes in the data stream, and
terminates lines of text (records) with CRLF as in ASCII text mode.  The file
receiver translates from international transfer syntax into the format
demanded by the local system or application.  All of this occurs before Kermit
packet encoding by the sender, and after Kermit packet decoding by the
receiver.

ISO 2022 states that "at the beginning of information interchange, except
where the interchanging parties have agreed otherwise, all designations shall
be defined by use of the appropriate escape sequences, and the shift status