<COMMENT>(--------------------------------------------------------------------)

<CHAPTER>(Introduction\chapter_introduction)

<P> This document provides an technical overview of the WASD VMS HTTP server.
It is primarily written for use internal to WASD and assumes that perpsective
without apology. Any additional usage is subordinate to its role within WASD.
The software has been written expressly for supporting an intra-organisational
hypertext environment on a VMS platform. It too,
is unreservedly tailored to this purpose and the requirements of WASD. All
programs were designed only to specifically comply with the requirements of
Compaq C, within a Compaq TCP/IP Services for VMS environment. 

<P> The document <EMPHASIS>(assumes\BOLD) a basic understanding of the 
hypertext technologies and uses terms without explaining them (e.g. HTTP, 
HTML, URL, CGI, SSI, etc.)   The reader is refered to documents specifically
on these topics (these are often best consulted online, on the Internet WWW). 

<COMMENT>(HTML=
<P> Also see
<A HREF="/ht_root/doc/scripting/">WASD Scripting Environment</A>
for information on using CGI, ISAPI, OSU, etc. scripting, and
)

<COMMENT>(HTML/OFF)
<P> Also see <QUOTE>(WASD Scripting Environment)
for information on using CGI, ISAPI, OSU, etc. scripting, and

<COMMENT>(HTML=
<A HREF="/ht_root/doc/env/">WASD Hypertext Environment</A>
for information on using the WASD VMS hypertext facilities.
)

<COMMENT>(HTML/OFF)
<QUOTE>(WASD Hypertext Environment) for information on using the 
WASD VMS hypertext facilities.

<P> It is strongly suggested those using printed versions of this document 
also access the Hypertext version.  It provides online demonstrations of some 
concepts.
<COMMENT>(HTML/ON) 

<HEAD>(Objectives\head_objective)

<P> The primary impetus for an internal Web environment was a 1993 decision by
Division management to make as much information as possible, both
administrative and research, available online (to use the current
term <HELLIPSIS> an <EMPHASIS>(intranet)). Early experimentation with a
<EMPHASIS>(Gopher) implementation soon made way for the obvious advantages of
the emerging Web technology.

<P> It then became the objective of this author to make <EMPHASIS>(all) of our
systems' VMS-related resources available via HTTP and HTML, regardless of the
underlying data or storage format. An examination of the WASD package will
show that this objective is substantially achieved.

<HEAD>(Reasons For a Local HTTPd\head_local_httpd)

<P> Reasons for developing a local HTTP server are few but compelling:

<LIST>(UNNUMBERED)

<LE> The WASD (then HFRD) Web implementation began mid-1994 (October 1997
note).

<LE> It was prefered to support the hypertext environment on a VMS platform.
This is currently (1995) the most widely used and accessable environment
within WASD. 

<LE> Existing servers (and there are quite a few variations) are largely Unix 
based, although it is being supported (to a greater or lesses extent) across a 
wide range of platforms.  Ports to VMS, if they exist, are often in progress 
or half-baked, employing <EMPHASIS>(Unix)isms that don't translate elegantly 
to the VMS environment. 

<LE> The VMS version of the CERN server (3.0-6) was evaluated during mid-1994: 

<LIST>(UNNUMBERED\-)

<LE> It is not multi-threaded under VMS (i.e. cannot support concurrent 
clients).  For example, a lengthy search may delay other clients for 
unacceptable periods. 

<LE> Its performance was good with document transfers, but became poor when 
running a <EMPHASIS>(script). 

<LE> It is acknowleged in the release notes that it cannot handle a client 
cancelling a data transfer (a not-uncommon action).  This was confirmed 
experimentally.

<ENDLIST>

<LE> An early version of the OSU (DECthreads) server was evaluated via
documentation mid-1994. (now in 1998 <HELLIPSIS> I am reticent to make further
comment on now such a widely deployed package :^)

<LE> HyperText Transport Protocol, in its current form, is relatively simple 
to implement to the level required to support intra-Divisional requirements. 

<LE> <EMPHASIS>(As of December 1995\BOLD) the server has worked extremely 
well and has a number of facilities tailored for the VMS environment.  It can 
continue to be utilized until there are overwhelming reasons for implementing 
something else.

<LE> <EMPHASIS>(As of June 1997\BOLD) the server and its associated software
continues to evolve and provide a stable and effective VMS Web environment,
even with the advent of a small number of commercial VMS Web products.

<LE> <EMPHASIS>(As of October 1999\BOLD) the package is beginning to mature as
an HTTP/1.0 solution, providing not only a fast and stable server but an
increasingly extensive collection of applications and tools.

<LE> <EMPHASIS>(As of July 2002\BOLD) it continues to be refined and extended. 
A greater emphasis on <QUOTE>(commercial) functionality has occured over the
past couple of years.

<ENDLIST>

<COMMENT>(--------------------------------------------------------------------)

<CHAPTER>(HTTPd - Overview\chapter_httpd)

<P> The most fundamental component of the WASD VMS Hypertext Services 
environment is the <EMPHASIS>(HTTPd\BOLD), or HyperText Protocol Transport
Daemon, or HTTP server.  WASD has a single-process, multi-threaded,
asynchronous I/O design.

<HEAD>(General\head_features_general)

<LIST>(UNNUMBERED)

<LE> concurrent, multi-threaded client support

<LE> <QUOTE>(GET), <QUOTE>(HEAD), <QUOTE>(POST) and <QUOTE>(PUT) HTTP method
support

<LE> virtual services (servers)

<LE> concurrent IPv4 and IPv6 support (requires underlying TCP/IP support)

<LE> requests above a configurable limit can be queued (<QUOTE>(throttling))

<LE> enhanced privacy using Secure Sockets Layer (SSL) technology, including
OpenSSL Toolkit, WASD OpenSSL, and HP SSL (Secure Sockets Layer) for OpenVMS
Alpha, IA64 and (now - late 2003) VAX product 

<LE> serves ODS-2 and ODS-5 (EFS) volumes, as well as file names encoded
using  the PATHWORKS 4/5, Advanced Server (PATHWORKS 6) and SRI (MultiNet NFS,
etc.) schemas

<LE> versatile directory listing (generic and VMS-style)

<LE> Server-Side Includes (HTML pre-processing)

<LE> configurable cache, with time-based and forced revalidation (reload)

<LE> byte-range support with 206 partial responses (useful for PDF and
restarting file download by modern browsers)

<LE> proxy serving, with local caching for all HTTP/1.0 methods, plus the
CONNECT method (allowing SSL tunnelling), along with FTP proxy (non-caching)

<LE> gatewaying between protocols (HTTP-to-SSL, SSL-to-HTTP, HTTP-to-FTP, etc.)

<LE> clickable-image support (both NCSA and CERN formats)

<ENDLIST> 

<HEAD>(Scripting\head_features_scripting)

<LIST>(UNNUMBERED)

<LE> CGI-compliant scripting

<LE> non-server and user account scripting

<LE> <QUOTE>(CGIplus) scripting
(offering reduced latency, increased throughput and reduced system impact)

<LE> <QUOTE>(Persistent) scripting,
Run-Time Environments (RTEs) that provide for simple persistent scripting

<LE> <QUOTE>(ISAPI) extensions/scripting
(also offering reduced latency, increased throughput and reduced system impact)

<LE> DECnet-based CGI scripting (with connection reuse)

<LE> OSU (DECthreads server) scripting emulation, with connection reuse
(as per OSU 3.3a), allowing many OSU scripts to be employed unmodified

<LE> script processor (e.g. PERL) configurable on file type (suffix)

<LE> configurable, automatic, MIME content-type initiated scripting
(<QUOTE>(presentation) scripting)

<ENDLIST>

<HEAD>(Access Control\head_features_access)

<LIST>(UNNUMBERED)

<LE> host-level, on per-host or per-domain

<LE> <QUOTE>(Basic) and <QUOTE>(Digest) user authentication and
path/group-based authorization

<LE> WASD-specific user databases

<LE> SYSUAF-authentication and VMS user security profile based file access
control

<LE> ACME service authentication (on applicable platforms)

<LE> X.509 client certificate authentication (for SSL transactions)

<LE> RFC1413 (<EMPHASIS>(ident) daemon) <QUOTE>(authentication)

<ENDLIST>

<HEAD>(Administration\head_features_admin)

<LIST>(UNNUMBERED)

<LE> multiple <EMPHASIS>(instances) (server processes) executing on the one
system allow continuous availability via rolling restarts and
<QUOTE>(fail-through) processing

<LE> <QUOTE>(one-button) control of multiple <EMPHASIS>(instances) on both
single systems and across clusters

<LE> online server configuration, including reports on requests,
loaded configuration, mapping rules, authorization information and graphical
activity displays

<LE> online, live server processing event report (WATCH)

<LE> Web-standard, <QUOTE>(common) and <QUOTE>(combined) access log formats
(allowing processing by most log-analysis tools), along with a user-definition
capability allowing custom log formats

<LE> logging periods, where log files automatically change on a daily,
weekly or monthly basis (keeps log files ordered and at a managable size)

<LE> customizable message database
(capable of supporting non-English and concurrent, multiple languages)

<ENDLIST> 

<HEAD1>(Server Behaviour\head_httpd_behaviour)

<P> The HTTPd executes permanently on the server host, listening for client
connection requests on TCP/IP port 80 (by default). It provides concurrent
services for a (technically) unlimitted number of clients (constrained only by
the server resources). When a client connects the server performs the
following tasks: 

<LIST>(NUMBERED)

<LE> creates a <EMPHASIS>(thread) for this request

<LE> reads and analyzes the HTTP request sent,
<LINE> depending on the nature of the request <HELLIPSIS>

<LIST>(UNNUMBERED)

<LE> transfer of a file (possibly from cache)

<LE> processing of an SSI HTML file

<LE> directory listing

<LE> file/directory create/update

<LE> server administration

<LE> processing of a clickable-image mapping file

<LE> create a process to to provide a CGI scripting environment:

<LIST>(UNNUMBERED\-)

<LE> standard CGI
 
<LE> WASD CGIplus

<LE> WASD Run-Time Environment

<ENDLIST> 

<LE> connects to a DECnet object to provide a scripting environment:

<LIST>(UNNUMBERED\-)

<LE> standard CGI
 
<LE> OSU (DECthreads) emulated

<ENDLIST> 

<ENDLIST> 

<LE> closes the connection to the client and disposes of the thread data 
structures 

<ENDLIST>

<P> For I/O intensive activities like file transfer and directory listing, the 
AST-driven code provides an efficient, multi-threaded environment for the 
concurrent serving of multiple clients. 

<P> For scripts, the technique of using multi-threaded, concurrent, spawned
subprocesses, detached autonomous processes, or network processes, attached to
standard input/output streams,  provides a versatile, extensible, powerful
scripting environment. Any DCL procedure or image executing within the process
can behave essentially as an HTTP server. This capability is employed to easily
extend the basic services provided by the core daemon code. An HTTP
script/server for this environment does not need to concern itself with network
activities, it merely reads and writes from the standard I/O streams. 

<HEAD1>(VMS Versions\head_vms_versions)

<P> The WASD server is officially supported on any VMS version from V6.0
upwards, on Alpha, IA64 (Itanium) and VAX architectures.  Obviously no
guarantees can be made for yet-to-be-released versions but at a worst-case
these should only require relinking.  Pre 7.<EMPHASIS>(n) versions of WASD have
also been known to compile and run successfully under V5.5-<EMPHASIS>(n)
(<REFERENCE>(head_version_552)).

<P> WASD on IA64 based systems has been initially ported and tested against
<QUOTE>HP OpenVMS Industry Standard 64 Evaluation Release Version
8.1<ENDQUOTE>.  Later VMS releases on this platform may require recompilation. 
No significant code changes were required for this port (as might be expected),
mainly requirements for Alpha specific conditional compilation expanded to
include IA64.

<P> Non-server account scripting requires a minimum VMS V6.2, to
provide the $PERSONA services required for this functionality.  Equivalent
functionality on earlier versions of VAX VMS (i.e. 6.0 and 6.1) is available
using the PERSONA_MACRO build option (<REFERENCE>(head_version_6061)).

<P> The WASD distribution and package organisation fully supports
mixed-architecture clusters (Alpha, IA64 and/or VAX in the one cluster) as one
integrated installation.

<HEAD1>(TCP/IP Packages\head_tcpip_packages)

<P> The WASD server uses the Compaq TCP/IP Services (UCX) BG $QIO interface. 
The following packages support this interface and may be used.

<LIST>(UNNUMBERED)

<LE> Compaq TCP/IP Services for OpenVMS (Hewlett Packard Corporation), any
version 

<LE> Digital TCP/IP Services for OpenVMS (aka UCX), any version

<LE> MultiNet for OpenVMS (Process Software Corporation), any version

<LE> TCPware (Process Software Corporation), any version

<ENDLIST>

<P> To deploy IPv6 services this package must support IPv6 (needless-to-say).

<HEAD1>(International Features\head_international)

<P> WASD provides a number of features that assist in the support of
non-English and multi-language sites.  These <QUOTE>(international) features
only apply to the server, not necessarily to any scripts!

<LIST>(UNNUMBERED)

<LE><EMPHASIS>(Language Variants\BOLD)

<P> A directory may contain language-specific variants of a basic document. 
When requesting the basic document name these variants are automatically
and transparently provided as the response if one matches preferences expresses
in the request's <QUOTE>(Accept-Language:) request header field.  Both text and
non-text documents (e.g. images) may be provided using this mechanism.

<P> Configuration information is provided in <REFERENCE>(head_language_variants).

<LE><EMPHASIS>(Character Sets\BOLD)

<P> Generally the default character set for documents on the Web is ISO-8859-1
(Latin-1). The server allows the specification of any character set as a
default for text document responses (plain and HTML).  In addition, text
document file types may be modified or additional ones specified that have a
different character set associated with that type. Furthermore, specific
character sets may be associated with mapping paths.  A site can therefore
relatively easily support multiple character set document resources.

<P> In addition the server may be configured to dynamically convert one
character set to another during request processing.  This is supported using
the VMS standard NCS character set conversion library.

<P> For further information see [CharsetDefault], [CharsetConvert] and
[AddType] in <REFERENCE>(chapter_config_directives).

<LE><EMPHASIS>(Server Messages\BOLD)

<P> The server uses an administrator-customizable database of messages that
can contain multiple language instances of some or all messages, using the
Latin-1 character set (ISO8859-1). Although the base English messages can be
completely changed and/or translated to provide any message text required or
desired, a more convenient approach is to supplement this base set with a
language-specific one.

<P> One language is designated the prefered language. This would most commonly
be the language appropriate to the geographical location and/or clientele of
the server. Another language is designated the base language. This must have a
complete set of messages and is a fall-back for any messages not configured
for the additional language. Of course this base language would most commonly
be the original English version.

<P> More than just two languages can be supported. If the browser has
<EMPHASIS>(prefered languages) set the server will attempt to match a message
with a language in this preference list. If not, then the server-prefered and
then the base language message would be issued, in that order. In this way it
would be possible to simultaneously provide for English, French, German and
Swedish audiences, just for example.

<P> For message configuration information see <REFERENCE>(chapter_msg_config).

<LE><EMPHASIS>(Server Dates\BOLD)

<P> Dates appearing in server-generated, non-administrative content (e.g.
directory listings, not META-tags, which use Web-standard time formats) will
use the natural language specified by any SYS$LANGUAGE environment in use on
the system or specifically created for the server.

<LE><EMPHASIS>(Virtual Services\BOLD)

<P> Virtual-server-associated mapping, authorization and character-sets allow
for easy multiple language and environment sites.  Further per-request
tailoring may be deployed using conditional rule mapping described below.
Single server can support multi-homed (host name) and multiple port services.

<P> For virtual services information see <REFERENCE>(chapter_config).

<LE><EMPHASIS>(Conditional Rule Mapping\BOLD)

<P> Mapping rules map requested URL paths to physical or other paths
(<REFERENCE>(chapter_mapping_rule)).  Conditional rules are only applied if the 
request matches criteria such as prefered language, host address (hence
geographical location to a certain extent), etc.  This allows requests for
generic documents (e.g. home pages) to be mapped to language versions
appropriate to the above criteria.

<P> For conditional mapping information see
<REFERENCE>(head_mapping_conditional).

<ENDLIST>

<COMMENT>(--------------------------------------------------------------------)

<HEAD1>(HTTP Methods Usage\head_http_method)

<P> This section describes WASD-specific characteristics of the available
HTTP/1.0 request methods.

<HEAD2>(GET\head_http_method_get)

<P> Of course, the GET method is used to access documents supplied by the
server.  There is nothing WASD server-specific about this method.

<HEAD2>(POST & PUT\head_http_method_post_put)

<P> The WASD HTTPd does not differentiate between POST and PUT methods, both
are handled identically.

<HEAD>(Script Handling\head_httpd_method_script)

<P> The <QUOTE>(normal) usage of the POST method is to return data from a
<LITERAL>(<FORM>..</FORM>) construct to a script running on the server.  In
this regard WASD is no different to other any web server; the form data is
delivered to the script's standard input as a stream of URL-encoded text.  For
example:

<CODE_EXAMPLE>
  name=Fred+Nurk&address=Fred%27s+House%0D%0A0+Nowhere+Lane&submit=Submit
<ENDCODE_EXAMPLE>

<P> Note that WWW_CONTENT_LENGTH will be the length of the form data.  See
<QUOTE>(Scripting Environment) document for further information.

<HEAD>(File Creation/Upload\head_httpd_method_file)

<P> If the client sends data back to the server using either the POST or
the PUT methods, without a script being mapped to be executed in response to
that data, the WASD HTTPd will create a file corresponding to the specified
path.  The data stream may be text or binary.

<P> <EMPHASIS>(Of course, for the server to accept POST and PUT data in this
manner, authentication and authorization must be enabled and allow such access
to the request path.\BOLD)

<P> The data stream is processed according to MIME content-type:

<LIST>(UNNUMBERED)

<LE><EMPHASIS>(application/x-www-form-urlencoded\BOLD)

<P> The server specially processes <QUOTE>(application/x-www-form-urlencoded)
POSTS (i.e. those generated by <LITERAL>(<FORM>...</FORM>), allowing files to
be created directly from HTML forms.  The processing eliminates any field names
from the URL-encoded data stream, placing only field values into the file. 
This capability can be quite useful and is demonstrated in the Update HTTPd
module

<COMMENT>(....................................................................)

<COMMENT>(HTML=
(<A HREF="/ht_root/src/httpd/upd.c">UPD.C</A>).
)

<COMMENT>(HTML/OFF)
<P><BOX>(online hypertext link).
<COMMENT>(HTML/ON)

<COMMENT>(....................................................................)

<LE><EMPHASIS>(multipart/form-data\BOLD)

<P> This server can process a request body according to RFC-1867,
<QUOTE>(Form-based File Upload in HTML).  As yet it is not a full
implementation.  It will not process  "multipart/mixed" subsections. The
implementation is basic, providing a facility to allow the upload of a file
into the server administered file system.  The ACTION= parameter of the
<LITERAL>(<FORM> tag) must specify the directory (as a URL path) in which the
uploaded file will be created.

<P> The following example HTML illustrates how a form may be used to upload a
file from the browser host file system:

<CODE_EXAMPLE>
  <LITERAL>(
  <FORM METHOD=POST ACTION="/web/directory/" ENCTYPE="multipart/form-data">
  <INPUT TYPE=submit VALUE=" Upload document ... ">
  <INPUT TYPE=file SIZE=50 NAME=uploadfile>
  </FORM>
)
<ENDCODE_EXAMPLE>

<COMMENT>(....................................................................)

<COMMENT>(HTML=
<P> Again, the <B>Upd</B>ate module
(<A HREF="/ht_root/src/httpd/upd.c">UPD.C</A>)
illustrates the use of this facility.
)

<COMMENT>(HTML/OFF)
<P><BOX>(online hypertext link)
<COMMENT>(HTML/ON)

<COMMENT>(....................................................................)

<NOTE>
This capability has only been tested against Netscape Navigator versions 2 and
3.  VMS Netscape Navigator 3.0b5 hangs if an upload of a variable-record format
file is attempted.  Stick to STREAM-LF or fixed, or convert the file to 
STREAM-LF.
<ENDNOTE>

<LE><EMPHASIS>(text/html\BOLD)
<LINE><EMPHASIS>(text/plain\BOLD)

<P> Text file created according to the path, VMS record type is STREAM-LF.

<LE><EMPHASIS>(image/gif\BOLD)
<LINE><EMPHASIS>(application/octet-stream\BOLD)
<LINE><EMPHASIS>(etc., etc., etc.\BOLD)

<P> Any other MIME type is considered <EMPHASIS>(binary) and the created file
is made an UNDEFINED record type.

<ENDLIST>

<HEAD>(Directory Creation\head_httpd_method_dir)

<P> A directory will be created by the HTTPd if a directory path is provided
with the POST or PUT methods.  For example:

<CODE_EXAMPLE>
  /dir1/dir2/dir-to-be-created/
<ENDCODE_EXAMPLE>

<HEAD>(File Deletion\head_httpd_method_file_del)

<P> A file will be deleted by the HTTPd if the file path ending with a
wildard version specification is provided with the POST or PUT methods.  For
example:

<CODE_EXAMPLE>
  /dir1/dir2/file-to-be.deleted;*
<ENDCODE_EXAMPLE>

<HEAD>(Directory Deletion\head_httpd_method_dir_del)

<P> A directory will be deleted by the HTTPd if a directory path ending with a
wildard version specification is provided with the POST or PUT methods.  For
example:

<CODE_EXAMPLE>
  /dir1/dir2/dir-to-be-deleted/;*
<ENDCODE_EXAMPLE>

<HEAD2>(DELETE\head_http_method_delete)

<P> The DELETE method should delete the file or directory corresponding to the
supplied path.

<COMMENT>(--------------------------------------------------------------------)
