<COMMENT>(--------------------------------------------------------------------)

<CHAPTER>(Introduction\chapter_introduction)

<P> This document provides an technical overview of the WASD VMS HTTP server.
It is primarily written for use internal to WASD and assumes that perspective
without apology. Any additional usage is subordinate to its role within WASD.
The software has been written expressly for supporting an intra-organisational
hypertext environment on a VMS platform (June 1997 note: since first writing
this the industry has settled on the term <EMPHASIS>(intranet\BOLD)). It too,
is unreservedly tailored to this purpose and the requirements of WASD. All
programs were designed only to specifically comply with the requirements of
VAX C and DEC C, within a DEC TCP/IP Services for VMS environment. 

<P> The document <EMPHASIS>(assumes\BOLD) a basic understanding of the 
hypertext technologies and uses terms without explaining them (e.g. HTTP, 
HTML, URL, CGI, SSI, etc.)   The reader is refered to documents specifically
on these topics (these are often best consulted on-line, on the Internet WWW). 

<COMMENT>(HTML=
<P> Also see
<A HREF="/ht_root/doc/scripting/">WASD Scripting Environment</A>
for information on using CGI, ISAPI, OSU, etc. scripting, and
)

<COMMENT>(HTML/OFF)
<P> Also see <QUOTE>(WASD Scripting Environment)
for information on using CGI, ISAPI, OSU, etc. scripting, and

<COMMENT>(HTML=
<A HREF="/ht_root/doc/env/">WASD Hypertext Environment</A>
for information on using the WASD VMS hypertext facilities.
)

<COMMENT>(HTML/OFF)
<QUOTE>(WASD Hypertext Environment) for information on using the 
WASD VMS hypertext facilities.

<P> It is strongly suggested those using printed versions of this document 
also access the Hypertext version.  It provides online demonstrations of some 
concepts.
<COMMENT>(HTML/ON) 

<HEAD>(Objectives\head_objective)

<P> The primary impetus for an internal Web environment was a 1993 decision by
Division management to make as much information as possible, both
administrative and research, internally available on-line (to use the current
term <HELLIPSIS> an <EMPHASIS>(intranet)). Early experimentation with a
<EMPHASIS>(Gopher) implementation soon made way for the obvious advantages of
the emerging Web technology.

<P> It then became the objective of this author to make <EMPHASIS>(all) of our
systems' VMS-related resources available via HTTP and HTML, regardless of the
underlying data or storage format. An examination of the WASD package will
show that this objective is substantially achieved.

<HEAD>(Reasons For a Local HTTPd\head_local_httpd)

<P> Reasons for developing a local HTTP server are few but compelling:

<LIST>(UNNUMBERED)

<LE> The WASD (then HFRD) Web implementation began mid-1994 (October 1997
note).

<LE> It was prefered to support the hypertext environment on a VMS platform.
This is currently (1995) the most widely used and accessable environment
within WASD. 

<LE> Existing servers (and there are quite a few variations) are largely Unix 
based, although it is being supported (to a greater or lesses extent) across a 
wide range of platforms.  Ports to VMS, if they exist, are often in progress 
or half-baked, employing <EMPHASIS>(Unix)isms that don't translate elegantly 
to the VMS environment. 

<LE> The VMS version of the CERN server (3.0-6) was evaluated during mid-1994: 

<LIST>(UNNUMBERED\-)

<LE> It is not multi-threaded under VMS (i.e. cannot support concurrent 
clients).  For example, a lengthy search may delay other clients for 
unacceptable periods. 

<LE> Its performance was good with document transfers, but became poor when 
running a <EMPHASIS>(script). 

<LE> It is acknowleged in the release notes that it cannot handle a client 
cancelling a data transfer (a not-uncommon action).  This was confirmed 
experimentally.

<ENDLIST>

<LE> An early version of the OSU (DECthreads) server was evaluated via
documentation mid-1994. (now in 1998 <HELLIPSIS> I am reticent to make further
comment on now such a widely deployed package :^)

<LE> HyperText Transport Protocol, in its current form, is relatively simple 
to implement to the level required to support intra-Divisional requirements. 

<LE> <EMPHASIS>(As of December 1995\BOLD) the server has worked extremely 
well and has a number of facilities tailored for the VMS environment.  It can 
continue to be utilized until there are overwhelming reasons for implementing 
something else.

<LE> <EMPHASIS>(As of June 1997\BOLD) the server and its associated software
continues to evolve and provide a stable and effective VMS Web environment,
even with the advent of a small number of commercial VMS Web products.

<LE> <EMPHASIS>(As of October 1999\BOLD) the package is beginning to mature as
an HTTP/1.0 solution, providing not only a fast and stable server but an
increasingly extensive collection of applications and tools.

<ENDLIST>

<COMMENT>(--------------------------------------------------------------------)

<CHAPTER>(HyperText Transport Protocol Daemon - Overview\chapter_httpd)

<P> The most fundamental component of the WASD VMS Hypertext Services 
environment is the <EMPHASIS>(HTTPd\BOLD), or HyperText Protocol Transport
Daemon, or HTTP server.  It has a single-process, multi-threaded, asynchronous
I/O design.

<HEAD1>(Server Features\head_httpd_features)

<P> The package provides a complete implementation of a basic HTTP/1.0 server,
including: 

<LIST>(UNNUMBERED)

<LE> concurrent, multi-threaded client support

<COMMENT>(HTML=<P>)

<LE> <QUOTE>(GET), <QUOTE>(HEAD), <QUOTE>(POST) and <QUOTE>(PUT) HTTP method
support

<COMMENT>(HTML=<P>)

<LE> virtual services (servers)

<COMMENT>(HTML=<P>)

<LE> optional enhanced privacy using Secure Sockets Layer (SSL) technology (USA
domestic, full-strength encryption)

<COMMENT>(HTML=<P>)

<LE> proxy serving, with local caching, for all HTTP/1.0 methods, plus the
CONNECT method (allowing SSL tunnelling)

<COMMENT>(HTML=<P>)

<LE> <QUOTE>(If-Modified-Since:) / <QUOTE>(304 Not Modified) functionality 
(document is only sent if modified since time specified by client)

<COMMENT>(HTML=<P>)

<LE> HTTP/1.0 de-facto persistent connections (i.e. <QUOTE>(Keep-Alive:),
reducing the number of TCP/IP connects required)

<COMMENT>(HTML=<P>)

<LE> versatile directory listing (generic and VMS-style)

<COMMENT>(HTML=<P>)

<LE> CGI-compliant subprocess scripting
(with configurable, automatic, MIME content-type initiated activation)

<COMMENT>(HTML=<P>)

<LE> non-server and user account scripting
(requires a minimum of VMS V6.2)

<COMMENT>(HTML=<P>)

<LE> <QUOTE>(CGIplus) scripting
(offering reduced latency, increased throughput and reduced system impact)

<COMMENT>(HTML=<P>)

<LE> <QUOTE>(Persistent) scripting,
Run-Time Environments (RTEs) that provide for simple persistent scripting
(with example Perl engine)

<COMMENT>(HTML=<P>)

<LE> <QUOTE>(ISAPI) extensions/scripting
(also offering reduced latency, increased throughput and reduced system impact)

<COMMENT>(HTML=<P>)

<LE> DECnet-based CGI scripting (with connection reuse)

<COMMENT>(HTML=<P>)

<LE> OSU (DECthreads server) scripting emulation, with connection reuse
(as per OSU 3.3a), allowing many OSU scripts to be employed unmodified

<COMMENT>(HTML=<P>)

<LE> script processor (e.g. PERL) configurable on file type (extension)

<COMMENT>(HTML=<P>)

<LE> Server-Side Includes (HTML pre-processing)

<COMMENT>(HTML=<P>)

<LE> configurable cache, with time-based and forced re-validation (reload)

<COMMENT>(HTML=<P>)

<LE> clickable-image support (both NCSA and CERN formats)

<COMMENT>(HTML=<P>)

<LE> <EMPHASIS>(smart) rule mapping (conditional rules)

<COMMENT>(HTML=<P>)

<LE> <QUOTE>(Basic) and <QUOTE>(Digest) user authentication and
path/group-based authorization

<COMMENT>(HTML=<P>)

<LE> SYSUAF-authentication and VMS user security profile based file access
control

<COMMENT>(HTML=<P>)

<LE> Web-standard, <QUOTE>(common) and <QUOTE>(combined) access log formats
(allowing processing by most log-analysis tools), along with a user-definition
capability allowing custom log formats

<COMMENT>(HTML=<P>)

<LE> logging periods, where log files automatically change on a daily,
weekly or monthly basis (keeps log files ordered and at a managable size)

<COMMENT>(HTML=<P>)

<LE> host-level access control, on per-host or per-domain acceptance or 
rejection 

<COMMENT>(HTML=<P>)

<LE> single server can support multi-homed (host name) and multiple port
services

<COMMENT>(HTML=<P>)

<LE> customizable message database
(capable of supporting non-English and concurrent, multiple languages)

<COMMENT>(HTML=<P>)

<LE> on-line server configuration, including reports on requests,
loaded configuration, mapping rules, authorization information and graphical
activity displays

<COMMENT>(HTML=<P>)

<LE> <QUOTE>(single button) control of multiple servers on both single systems
and clusters 

<COMMENT>(HTML=<P>)

<LE> on-line, live server processing event report (WATCH)

<ENDLIST> 

<HEAD1>(Server Behaviour\head_httpd_behaviour)

<P> The HTTPd executes permanently on the server host, listening for client
connection requests on TCP/IP port 80 (by default). It provides concurrent
services for a (technically) unlimitted number of clients (constrained only by
the server resources). When a client connects the server performs the
following tasks: 

<LIST>(NUMBERED)

<LE> creates a <EMPHASIS>(thread) for this request

<LE> reads and analyzes the HTTP request sent,
<LINE> depending on the nature of the request <HELLIPSIS>

<LIST>(UNNUMBERED)

<LE> transfer of a file (possibly from cache)

<LE> processing of an SSI HTML file

<LE> processing of a clickable-image mapping file

<LE> directory listing

<LE> file/directory create/update

<LE> server administration

<LE> web file-system update

<LE> interpretation of a menu file

<LE> spawn a subprocess to to provide a CGI scripting environment:

<LIST>(UNNUMBERED\-)

<LE> standard CGI
 
<LE> WASD CGIplus

<ENDLIST> 

<LE> connects to a DECnet object to provide a scripting environment:

<LIST>(UNNUMBERED\-)

<LE> standard CGI
 
<LE> OSU (DECthreads) emulated

<ENDLIST> 

<ENDLIST> 

<LE> closes the connection to the client and disposes of the thread data 
structures 

<ENDLIST>

<P> For I/O intensive activities like file transfer and directory listing, the 
AST-driven code provides an efficient, multi-threaded environment for the 
concurrent serving of multiple clients. 

<P> For scripts, the technique of using multi-threaded, concurrent, spawned
subprocesses or network processes, attached to standard input/output streams,
provides a versatile, extensible, powerful scripting environment. Any DCL
procedure or image executing within the process can behave essentially as an
HTTP server. This capability is employed to easily extend the basic services
provided by the core daemon code. An HTTP script/server for this environment
does not need to concern itself with network activities, it merely reads and
writes from the standard I/O streams. 

<HEAD1>(VMS Versions\head_vms_versions)

<P> The WASD server is officially supported on any VMS version from V6.0
upwards, on both Alpha and VAX architectures.  Obviously no guarantees can be
made for yet-to-be-released versions but at a worst-case these should only
require relinking.  WASD has also been known to compile and run successfully
under V5.5-2 (see <REFERENCE>(head_version_552)).

<P> Note that the non-server account scripting requires a minimum VMS V6.2, to
support the $PERSONA services required for this functionality.

<P> The WASD distribution and package organisation fully supports
mixed-architecture clusters (both Alpha and VAX in the one cluster) as one
integrated installation.

<HEAD1>(TCP/IP Packages\head_tcpip_packages)

<P> The WASD server uses the Compaq TCP/IP Services (UCX) BG $QIO interface. 
The following packages support this interface and may be used.

<LIST>(UNNUMBERED)

<LE> Compaq TCP/IP Services for OpenVMS (Compaq Computer Corporation), any
version 

<LE> Digital TCP/IP Services for OpenVMS (aka UCX), any version

<LE> MultiNet for OpenVMS (Process Software Corporation), any version

<LE> TCPware (Process Software Corporation), any version

<ENDLIST>

<HEAD1>(International Features\head_international)

<P> As of version 4.4 the HTTP server has undergone a limited degree of
<EMPHASIS>(internationalization).  The impetus for this development was the use
of the package in some European countries where English is not the first
language.

<P> As this was not within the original critera while building the server it
has meant an extensive rewrite of some functionality within the code (I hope
it's appreciated ;^) The objective is to provide users of the server with a
more familiar and therefore more comfortable environment. Unfortunately the
server administrators are still confronted by an English-language interface
and documentation, but then the package doesn't pretend to be
<EMPHASIS>(international software)!

<HEAD>(Aspects\head_international_aspects)

<P> The international features only apply to the server, not necessarily to any
scripts!

<LIST>(UNNUMBERED)

<LE><EMPHASIS>(Server Messages\BOLD)

<P> The server uses an administrator-customizable database of messages that
can contain multiple language instances of some or all messages, using the
Latin-1 character set (ISO8859-1). Although the base English messages can be
completely changed and/or translated to provide any message text required or
desired, a more convenient approach is to supplement this base set with a
language-specific one.

<P> One language is designated the prefered language. This would most commonly
be the language appropriate to the geographical location and/or clientele of
the server. Another language is designated the base language. This must have a
complete set of messages and is a fall-back for any messages not configured
for the additional language. Of course this base language would most commonly
be the original English version.

<P> More than just two languages can be supported. If the browser has
<EMPHASIS>(prefered languages) set the server will attempt to match a message
with a language in this preference list. If not, then the server-prefered and
then the base language message would be issued, in that order. In this way it
would be possible to simultaneously provide for English, French, German and
Swedish audiences, just for example.

<P> For message configuration information see <REFERENCE>(chapter_msg_config).

<COMMENT>(HTML=<P>)

<LE><EMPHASIS>(Server Dates\BOLD)

<P> Dates appearing in server-generated, non-administrative content (e.g.
directory listings, not META-tags, which use Web-standard time formats) will
use the natural language specified by any SYS$LANGUAGE environment in use on
the system or specifically created for the server.

<COMMENT>(HTML=<P>)

<LE><EMPHASIS>(Virtual Services\BOLD)

<P> Virtual-server-associated mapping, authorization and character-sets allow
for easy multiple language and environment sites.  Further per-request
tailoring may be deployed using conditional rule mapping described below.

<P> For virtual services information see <REFERENCE>(chapter_config).

<COMMENT>(HTML=<P>)

<LE><EMPHASIS>(Conditional Rule Mapping\BOLD)

<P> Mapping rules map requested URL paths to physical or other paths (see
<REFERENCE>(chapter_mapping_rule)).  Conditional rules are only applied if the
request matches criteria such as prefered language, host address (hence
geographical location to a certain extent), etc.  This allows requests for
generic documents (e.g. home pages) to be mapped to language versions
appropriate to the above criteria.

<P> For conditional mapping information see
<REFERENCE>(head_mapping_conditional).

<COMMENT>(HTML=<P>)

<LE><EMPHASIS>(Character Sets\BOLD)

<P> Generally the default character set for documents on the Web is ISO-8859-1
(Latin-1). The server allows the specification of any character set as a
default for text document responses (plain and HTML).  In addition, text
document file types may be modified or additional ones specified that have a
different character set associated with that type. Furthermore, specific
character sets may be associated with mapping paths.  A site can therefore
relatively easily support multiple character set document resources.

<P> Major WASD scripts also use the server default character set and/or any
request determined character set (based on request path with conditional
mapping to language or character set preferences) via CGI variable contents.

<P> For further information see
[CharsetDefault] and [AddType] in <REFERENCE>(head_set_rules).

<ENDLIST>

<COMMENT>(--------------------------------------------------------------------)

<HEAD1>(HTTP Methods Usage\head_http_method)

<P> This section describes WASD-specific characteristics of the available
HTTP/1.0 request methods.

<HEAD2>(GET\head_http_method_get)

<P> Of course, the GET method is used to access documents supplied by the
server.  There is nothing WASD server-specific about this method.

<HEAD2>(POST & PUT\head_http_method_post_put)

<P> The WASD HTTPd does not differentiate between POST and PUT methods, both
are handled identically.

<HEAD>(Script Handling\head_httpd_method_script)

<P> The <QUOTE>(normal) usage of the POST method is to return data from a
<LITERAL>(<FORM>..</FORM>) construct to a script running on the server.  In
this regard WASD is no different to other any web server; the form data is
delivered to the script's standard input as a stream of URL-encoded text.  For
example:

<CODE_EXAMPLE>
  name=Fred+Nurk&address=Fred%27s+House%0D%0A0+Nowhere+Lane&submit=Submit
<ENDCODE_EXAMPLE>

<P> Note that WWW_CONTENT_LENGTH will be the length of the form data.  See
<QUOTE>(Scripting Environment) document for further information.

<HEAD>(File Creation/Upload\head_httpd_method_file)

<P> If the client sends data back to the server using either the POST or
the PUT methods, without a script being mapped to be executed in response to
that data, the WASD HTTPd will create a file corresponding to the specified
path.  The data stream may be text or binary.

<P> <EMPHASIS>(Of course, for the server to accept POST and PUT data in this
manner, authentication and authorization must be enabled and allow such access
to the request path.\BOLD)

<P> The data stream is processed according to MIME content-type:

<LIST>(UNNUMBERED)

<LE><EMPHASIS>(application/x-www-form-urlencoded\BOLD)

<P> The server specially processes <QUOTE>(application/x-www-form-urlencoded)
POSTS (i.e. those generated by <LITERAL>(<FORM>...</FORM>), allowing files to
be created directly from HTML forms.  The processing eliminates any field names
from the URL-encoded data stream, placing only field values into the file. 
This capability can be quite useful and is demonstrated in the Update HTTPd
module

<COMMENT>(....................................................................)

<COMMENT>(HTML=
(<A HREF="/ht_root/src/httpd/upd.c">UPD.C</A>).
)

<COMMENT>(HTML/OFF)
<P><BOX>(online hypertext link).
<COMMENT>(HTML/ON)

<COMMENT>(....................................................................)

<COMMENT>(HTML=<P>)

<LE><EMPHASIS>(multipart/form-data\BOLD)

<P> This server can process a request body according to RFC-1867,
<QUOTE>(Form-based File Upload in HTML).  As yet it is not a full
implementation.  It will not process  "multipart/mixed" subsections. The
implementation is basic, providing a facility to allow the upload of a file
into the server administered file system.  The ACTION= parameter of the
<LITERAL>(<FORM> tag) must specify the directory (as a URL path) in which the
uploaded file will be created.

<P> The following example HTML illustrates how a form may be used to upload a
file from the browser host file system:

<CODE_EXAMPLE>
  <LITERAL>(
  <FORM METHOD=POST ACTION="/web/directory/" ENCTYPE="multipart/form-data">
  <INPUT TYPE=submit VALUE=" Upload document ... ">
  <INPUT TYPE=file SIZE=50 NAME=uploadfile>
  </FORM>
)
<ENDCODE_EXAMPLE>

<COMMENT>(....................................................................)

<COMMENT>(HTML=
<P> Again, the <B>Upd</B>ate module
(<A HREF="/ht_root/src/httpd/upd.c">UPD.C</A>)
illustrates the use of this facility.
)

<COMMENT>(HTML/OFF)
<P><BOX>(online hypertext link)
<COMMENT>(HTML/ON)

<NOTE>
This capability has only been tested against Netscape Navigator versions 2 and
3.  VMS Netscape Navigator 3.0b5 hangs if an upload of a variable-record format
file is attempted.  Stick to STREAM-LF or fixed, or convert the file to 
STREAM-LF.
<ENDNOTE>

<COMMENT>(HTML=<P>)

<LE><EMPHASIS>(text/html\BOLD)
<LINE><EMPHASIS>(text/plain\BOLD)

<P> Text file created according to the path, VMS record type is STREAM-LF.

<COMMENT>(HTML=<P>)

<LE><EMPHASIS>(image/gif\BOLD)
<LINE><EMPHASIS>(application/octet-stream\BOLD)
<LINE><EMPHASIS>(etc., etc., etc.\BOLD)

<P> Any other MIME type is considered <EMPHASIS>(binary) and the created file
is made an UNDEFINED record type.

<ENDLIST>

<HEAD>(Directory Creation\head_httpd_method_dir)

<P> A directory will be created by the HTTPd if a directory path is provided
with the POST or PUT methods.  For example:

<CODE_EXAMPLE>
  /dir1/dir2/dir-to-be-created/
<ENDCODE_EXAMPLE>

<HEAD>(File Deletion\head_httpd_method_file_del)

<P> A file will be deleted by the HTTPd if the file path ending with a
wildard version specification is provided with the POST or PUT methods.  For
example:

<CODE_EXAMPLE>
  /dir1/dir2/file-to-be.deleted;*
<ENDCODE_EXAMPLE>

<HEAD>(Directory Deletion\head_httpd_method_dir_del)

<P> A directory will be deleted by the HTTPd if a directory path ending with a
wildard version specification is provided with the POST or PUT methods.  For
example:

<CODE_EXAMPLE>
  /dir1/dir2/dir-to-be-deleted/;*
<ENDCODE_EXAMPLE>

<HEAD2>(DELETE\head_http_method_delete)

<P> The DELETE method should delete the file or directory corresponding to the
supplied path.

<COMMENT>(--------------------------------------------------------------------)
