/*****************************************************************************/
/*
                               Throttle.c

Request "throttling" is a term adopted to describe controlling the number of
concurrent requests that can be processing against any specified path at any
one time.  Requests in excess of this value are then FIFO queued up to an
optional limit waiting for a being-processed request to conclude allowing the
next queued request to continue processing again.  This is primarily intended
to limit concurrent resource-intensive script execution but could be applied to
*any* resource path.  Here's one dictionary description.

  throttle n 1: a valve that regulates the supply of fuel to the engine
  [syn: accelerator, throttle valve] 2: a pedal that controls the throttle
  valve; "he stepped on the gas" [syn: accelerator, accelerator pedal, gas
  pedal, gas, gun] v 1: place limits on; "restrict the use of this parking
  lot" [syn: restrict, restrain, trammel, limit, bound, confine] 2: squeeze
  the throat of; "he tried to strangle his opponent" [syn: strangle,
  strangulate] 3: reduce the air supply; of carburetors [syn: choke] 

This is applied to a path (or paths) using the mapping SET THROTTLE= rule. 
This rule allows a maximum concurrent number of requests to be specified, and
optionally a maximum number of queued requests.  When the maximum queued
requests is exceeded the client receives a 503 (server too busy) status.  Empty
or zero parameters may be included between the commas.

  throttle=from[,to,resume,busy,timeout-queue,timeout-busy]
  throttle=n1[,n2,n3,n4,to1,to2]

  o  from        (n1) concurrent requests before queuing begins
  o  to          (n2) queuing continues up to this value, when the queue FIFOs
  o  resume      (n3) FIFO continues to this value, where queuing begins again
  o  busy        (n4) absolute maximum concurrent requests before 503 "busy"
  o  t/o-queue   (to1) period before a queued request is processed
  o  t/o-busy    (to2) period before a queued request is 503 "busy"ed

When a throttle rule is loaded it is checked for "sensible" values.  Basically
this means that each successive value is larger than it's predecessor.


DESCRIPTION
-----------

  o  If 'from' (n1) is exceeded then begin to queue.  Actively processing
requests does not increase, queue length does.

  o  If 'to' (n2) is specified and exceeded then begin to FIFO requests from
the queue into processing.  Queue length does not increase (new are being put
onto the queue, previous being taken off of the other end of the queue), but
number processing now begins to increase.
  
  o  If 'resume' (n3) is specified this acts as an absolute control on all
request PROCESSING associated with the path.  After this value the number of
requests actively being processed does not increase but queue length again
does.
  
  o  If 'busy' (n4) is exceeded ALWAYS immediately generate a 503 "busy".

  o  If 'timeout-queue' (to1) is specified this causes queued requests
exceeding the period to be FIFOed from the queue into processing unless the
'resume' (n3) limit would be exceeded.  If this would be exceeded they remain
in the queue (potentially indefinitely, or until they FIFO off the queue, or
until timeout-busy (to2) occurs).
  
  o  If 'timeout-busy' (to2) is specified queued requests exceeding the period
are immediately terminated with a 503 "busy" status.  A 'timeout-busy' only
begins after the expiry of any 'timeout-queue'.
  

EXAMPLES
--------

1) throttle=10

Requests up to 10 are concurrently processed.  When 10 is reached further
requests are queued to server capacity.

2) throttle=10,20

Concurrent requests to 10 are processed immediately.  From 11 to 20 requests
are queued.  After 20 all requests are queued but also result in a request
FIFOing off the queue to be processed (queue length is static, number being
processed increases to server capacity).

3) throttle=15,30,40

Concurrent requests up to 15 are immediately processed.  Requests 16 through to
30 are queued, while 31 to 40 requests result in the new requests being queued
and waiting requests being FIFOed into processing.  Concurrent requests from 41
onwards are again queued, in this scenario to server capacity.

4) throttle=10,20,30,40                   

Concurrent requests up to 10 are immediately processed.  Requests 11 through
to 20 will be queued.  Concurrent requests from 21 to 30 are queued too, but at
the same time waiting requests are FIFOed from the queue (resulting in 10 (n1)
+ 10 (n3-n2) = 20 being processed).  From 31 onwards requests are just queued. 
Up to 40 concurrent requests may be against the path before all new requests
are immediately returned with a 503 "busy" status.  With this scenario no more
than 20 can be concurrently processed with 20 concurrently queued.

5) throttle=10,,,30

Concurrent requests up to 10 are processed.  When 10 is reached requests are
queued up to request 30.  When request 31 arrives it is immediately given a 503
"busy" status.

6) throttle=10,20,30,40,00:02:00

This is basically the same as scenario 4) but with a resume-on-timeout of two
minutes.  If there are currently 15 (or 22 or 28) requests (n1 exceeded, n3
still within limit) the queued requests will begin processing on timeout. 
Should there be 32 processing (n3 has reached limit) the request will continue
to sit in the queue.  The timeout would not be reset.

6) throttle=15,30,40,,,00:03:00

This is basically the same as scenario 3) but with a busy-on-timeout of three
minutes.  When the timeout expires the request is immediately dequeued with a
503 "busy" status.


IMPLEMENTATION
--------------

Request throttling is implemented by the MAPURL.C rule loading providing the
SET THROTTLE= rule parameters against the path, AS WELL AS counting the number
of such paths in the rules.  This number is stored along with the maxima and is
used as an index into, as well as to set up, a dynamically allocated array of
structures used to support the concurrent usage tracking and queuing against
that particular path.  The final example specifies that there will be one
request processing, an unlimited number of queued requests, and once the queue
length exceeds five then multiple requests may be processed concurrently.

When path mapping SETs a throttle maximum against the request's path the
associated index number is used to select the corresponding element of the
usage structure array.  Simple and reasonably efficient.  RULE RELOADING could
present a problem with this schema ... but doesn't.  The array can grow in size
to accomodate a new rule load with additional throttles, but will never shrink. 
(being based on an array index, not a pointer makes this possible).  This
allows existing requests with buffered indices to (somewhat) correctly access
the array and be (somewhat) correctly processed.  If the actual paths
represented by array elements change there may be some "confusion" in what the
processing and queuing represents.  This could possibly result in some
resources temporarily being inappropriately throttled but this gradually
filters out of the functionality as associated requests conclude.  There are no
fatal implications (that I can see, anyway) in this scheme.


VERSION HISTORY
---------------
20-JUL-2003  MGD  revise reporting format
04-AUG-2001  MGD  support module WATCHing,
                  fix end throttle call to RequestExecutePostThrottle()
18-MAY-2001  MGD  bugfix; exceeding the point where we should start to FIFO
                  (jfp@altavista.com)
08-MAY-2001  MGD  modify throttle parameter meanings and functionality
08-APR-2001  MGD  add 'queue-length' to throttling
13-MAR-2001  MGD  initial development
*/
/*****************************************************************************/

#ifdef WASD_VMS_V6
#undef _VMS_V6_SOURCE
#define _VMS_V6_SOURCE
#undef __VMS_VER
#define __VMS_VER 60000000
#undef __CRTL_VER
#define __CRTL_VER 60000000
#endif

/* standard C header files */
#include <ctype.h>
#include <errno.h>
#include <math.h>
#include <stdio.h>
#include <string.h>

/* VMS related header files */
#include <descrip.h>
#include <ssdef.h>
#include <stsdef.h>

/* application header files */
#include "wasd.h"

#define WASD_MODULE "THROTTLE"

/******************/
/* global storage */
/******************/

int  ThrottleBusyMetricTotal,
     ThrottleBusyMetricTotal503,
     ThrottleTotal;

THROTTLE_STRUCT  *ThrottleArray;

/********************/
/* external storage */
/********************/

#ifdef DBUG
extern BOOL Debug;
#else
#define Debug 0 
#endif

extern char  ErrorSanityCheck[],
             ServerHostPort[],
             SoftwareID[],
             Utility[];

extern ACCOUNTING_STRUCT  *AccountingPtr;
extern CONFIG_STRUCT  Config;
extern MAPPING_META  *MappingMetaPtr;
extern MSG_STRUCT  Msgs;
extern WATCH_STRUCT  Watch;

/*****************************************************************************/
/*
(Re)initialize the global throttle structure array.
*/ 

ThrottleInit ()

{
   int  idx;
   THROTTLE_STRUCT  *tsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCH_MODULE(WATCH_MOD_THROTTLE))
      WatchThis (NULL, FI_LI, WATCH_MOD_THROTTLE, "ThrottleInit() !UL !UL",
                 ThrottleTotal, MappingMetaPtr->ThrottleTotal);

   if (ThrottleTotal < MappingMetaPtr->ThrottleTotal)
      ThrottleTotal = MappingMetaPtr->ThrottleTotal;

   if (!ThrottleTotal) return (SS$_NORMAL);

   ThrottleArray = (THROTTLE_STRUCT*)
      VmRealloc (ThrottleArray, ThrottleTotal*sizeof(THROTTLE_STRUCT), FI_LI);

   /* in case this is a mapping rule reload reset all the counters */
   ThrottleZero ();
}

/*****************************************************************************/
/*
Zero the accumulators associated with the throttle structure array.
*/ 

ThrottleZero ()

{
   int  idx;
   THROTTLE_STRUCT  *tsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCH_MODULE(WATCH_MOD_THROTTLE))
      WatchThis (NULL, FI_LI, WATCH_MOD_THROTTLE, "ThrottleZero()");

   for (idx = 0; idx < ThrottleTotal; idx++)
   {
      tsptr = &ThrottleArray[idx];
      tsptr->MaxProcessingCount =
         tsptr->MaxQueuedCount =
         tsptr->Total503Count =
         tsptr->TotalCount =
         tsptr->TotalFiFoCount =
         tsptr->TotalQueuedCount =
         tsptr->TotalTimeoutBusyCount =
         tsptr->TotalTimeoutQueueCount = 0;
   }

   ThrottleMonitorReset ();
}

/*****************************************************************************/
/*
Called by HttpdTick() each minute or when there is no more server activity.
*/ 

ThrottleMonitorReset ()

{
   int  idx;
   THROTTLE_STRUCT  *tsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCH_MODULE(WATCH_MOD_THROTTLE))
      WatchThis (NULL, FI_LI, WATCH_MOD_THROTTLE, "ThrottleMonitorReset()");

   InstanceMutexLock (INSTANCE_MUTEX_HTTPD);

   AccountingPtr->ThrottleBusyMetric =
      AccountingPtr->ThrottleCurrentlyProcessing =
      AccountingPtr->ThrottleCurrentlyQueued =
      ThrottleBusyMetricTotal =
      ThrottleBusyMetricTotal503 = 0;

   /* zeroing upsets the 'currently' data, recalculate just in case */
   for (idx = 0; idx < ThrottleTotal; idx++)
   {
      tsptr = &ThrottleArray[idx];
      AccountingPtr->ThrottleCurrentlyProcessing +=
         tsptr->CurrentlyProcessingCount;
      AccountingPtr->ThrottleCurrentlyQueued +=
         tsptr->CurrentlyQueuedCount;
   }

   InstanceMutexUnLock (INSTANCE_MUTEX_HTTPD);
}

/*****************************************************************************/
/*
The request path having a maximum concurrent request limit set has resulted in
a call to this function before actually processing the request.  If the request
is stalled by queueing then return SS$_RETRY, if it can continue immediately
then return SS$_CONTINUE.  See description at the beginning of this module for
an explanation of the algroithm used in this function.
*/ 

int ThrottleBegin (REQUEST_STRUCT *rqptr)

{
   BOOL  ProcessRequest,
         QueueRequest;
   char  *cptr;
   double  fScratch;
   THROTTLE_STRUCT  *tsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCHING(rqptr) && (WATCH_MODULE(WATCH_MOD_THROTTLE)))
      WatchThis (rqptr, FI_LI, WATCH_MOD_THROTTLE, "ThrottleBegin()");

   ThrottleBusyMetricTotal++;
   if (ThrottleBusyMetricTotal503)
   {
      /* must update the metric here to potentially reduce the metric */
      fScratch = (double)ThrottleBusyMetricTotal503 * 100.0 /
                 (double)ThrottleBusyMetricTotal;
      InstanceMutexLock (INSTANCE_MUTEX_HTTPD);
      AccountingPtr->ThrottleBusyMetric = (int)fScratch;
      if (modf (fScratch, &fScratch) >= 0.5)
         AccountingPtr->ThrottleBusyMetric++;
      InstanceMutexUnLock (INSTANCE_MUTEX_HTTPD);
   }

   tsptr = &ThrottleArray[rqptr->rqPathSet.ThrottleIndex];

   tsptr->TotalCount++;

   /* throttle=from,to,resume,busy,t/o-queue,t/o-busy */
   ProcessRequest = QueueRequest = false;

   /* if it can be processed immediately */
   if (tsptr->CurrentlyProcessingCount <
       rqptr->rqPathSet.ThrottleFrom)
      ProcessRequest = true;
   else
   /* if it can be queued */
   if (!rqptr->rqPathSet.ThrottleBusy ||
       tsptr->CurrentlyQueuedCount +
       tsptr->CurrentlyProcessingCount <
       rqptr->rqPathSet.ThrottleBusy)
   {
      /* queue it up, perhaps we'll also be FIFOing */
      QueueRequest = true;
      /* if exceeding the point where we should start to FIFO */
      if (rqptr->rqPathSet.ThrottleTo &&
          tsptr->CurrentlyQueuedCount >=
          rqptr->rqPathSet.ThrottleTo -
          rqptr->rqPathSet.ThrottleFrom)
      {
         /* if still under any limit imposed on FIFO processing */
         if (!rqptr->rqPathSet.ThrottleResume ||
             tsptr->CurrentlyProcessingCount <
             (rqptr->rqPathSet.ThrottleResume -
              rqptr->rqPathSet.ThrottleTo) +
              rqptr->rqPathSet.ThrottleFrom)
            ProcessRequest = true;
      }
   }

   if (WATCHING(rqptr) && WATCH_CATEGORY(WATCH_REQUEST))
   {
      /*********/
      /* watch */
      /*********/

      if (ProcessRequest && QueueRequest)
         cptr = "->QUEUE->";
      else
      if (ProcessRequest)
         cptr = "PROCESS";
      else
      if (QueueRequest)
         cptr = "->QUEUE";
      else
         cptr = "BUSY";

      WatchThis (rqptr, FI_LI, WATCH_REQUEST,
                 "THROTTLE set:!UL,!UL,!UL,!UL current:!UL,!UL !AZ",
                 rqptr->rqPathSet.ThrottleFrom,
                 rqptr->rqPathSet.ThrottleTo,
                 rqptr->rqPathSet.ThrottleResume,
                 rqptr->rqPathSet.ThrottleBusy,
                 tsptr->CurrentlyProcessingCount,
                 tsptr->CurrentlyQueuedCount,
                 cptr);
   }

   /* this queue code section must precede the process code section */
   if (QueueRequest)
   {
      /*********/
      /* queue */
      /*********/

      tsptr->TotalQueuedCount++;
      tsptr->CurrentlyQueuedCount++;
      InstanceGblSecIncrLong (&AccountingPtr->ThrottleCurrentlyQueued);
      if (tsptr->CurrentlyQueuedCount > tsptr->MaxQueuedCount)
         tsptr->MaxQueuedCount = tsptr->CurrentlyQueuedCount;

      /* the list entry data becomes a pointer to the request structure */
      rqptr->ThrottleListEntry.DataPtr = (void*)rqptr;

      /* add entry to the tail of the waiting list (FIFO) */
      ListAddTail (&tsptr->QueuedList, &rqptr->ThrottleListEntry);

      if (rqptr->rqPathSet.ThrottleTimeoutQueue ||
          rqptr->rqPathSet.ThrottleTimeoutBusy)
         HttpdTimerSet (rqptr, TIMER_THROTTLE, 0);
 
      /* if this was not a FIFO operation then return here */
      if (!ProcessRequest) return (SS$_RETRY);

      /****************/
      /* FIFO process */
      /****************/

      /* release the head of the queued requests (note reuse of 'rqptr') */
      rqptr = (REQUEST_STRUCT*)(tsptr->QueuedList.HeadPtr->DataPtr);
      ThrottleRelease (rqptr, true);

      return (SS$_RETRY);
   }

   /* this process code section must follow the queue code section */
   if (ProcessRequest)
   {
      /***********/
      /* process */
      /***********/

      tsptr->CurrentlyProcessingCount++;
      if (tsptr->CurrentlyProcessingCount > tsptr->MaxProcessingCount)
         tsptr->MaxProcessingCount = tsptr->CurrentlyProcessingCount;
      InstanceGblSecIncrLong (&AccountingPtr->ThrottleCurrentlyProcessing);

      return (SS$_CONTINUE);
   }

   /************/
   /* too busy */
   /************/

   /* remove this request's claim to fame */
   rqptr->rqPathSet.ThrottleIndex =
      rqptr->rqPathSet.ThrottleBusy =
      rqptr->rqPathSet.ThrottleFrom =
      rqptr->rqPathSet.ThrottleResume =
      rqptr->rqPathSet.ThrottleTo =
      rqptr->rqPathSet.ThrottleTimeoutBusy =
      rqptr->rqPathSet.ThrottleTimeoutQueue = 0;

   tsptr->Total503Count++;
   rqptr->rqResponse.HttpStatus = 503;
   ErrorGeneral (rqptr, MsgFor(rqptr,MSG_GENERAL_TOO_BUSY), FI_LI);

   /* update to increase the metric */
   ThrottleBusyMetricTotal503++;
   fScratch = (double)ThrottleBusyMetricTotal503 * 100.0 /
              (double)ThrottleBusyMetricTotal;
   InstanceMutexLock (INSTANCE_MUTEX_HTTPD);
   AccountingPtr->ThrottleBusyMetric = (int)fScratch;
   if (modf (fScratch, &fScratch) >= 0.5) AccountingPtr->ThrottleBusyMetric++;
   InstanceMutexUnLock (INSTANCE_MUTEX_HTTPD);

   return (SS$_ABORT);
}

/*****************************************************************************/
/*
The request path having a maximum concurrent request limit set has resulted in
a call to this function at the end of processing the request.  Adjust the
concurrent processing counter and check if there are any requests queued
waiting for processing.  If not just return.  If there is/are then remove the
front of the list (FIFO) and call the AST function address stored when it was
originally queued to continue processing.  See description above for further
detail on "throttling" request processing.
*/ 

ThrottleEnd (REQUEST_STRUCT *rqptr)

{
   int  status;
   THROTTLE_STRUCT  *tsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCHING(rqptr) && (WATCH_MODULE(WATCH_MOD_THROTTLE)))
      WatchThis (rqptr, FI_LI, WATCH_MOD_THROTTLE,
                 "ThrottleEnd() !&F", &ThrottleEnd);

   tsptr = &ThrottleArray[rqptr->rqPathSet.ThrottleIndex];

   if (WATCHING(rqptr) && WATCH_CATEGORY(WATCH_REQUEST))
      WatchThis (rqptr, FI_LI, WATCH_REQUEST,
                 "THROTTLE set:!UL,!UL,!UL,!UL current:!UL,!UL END",
                 rqptr->rqPathSet.ThrottleFrom,
                 rqptr->rqPathSet.ThrottleTo,
                 rqptr->rqPathSet.ThrottleResume,
                 rqptr->rqPathSet.ThrottleBusy,
                 tsptr->CurrentlyProcessingCount,
                 tsptr->CurrentlyQueuedCount);

   /* careful, these counters can get scrambled during a mapping reload */
   if (tsptr->CurrentlyProcessingCount) tsptr->CurrentlyProcessingCount--;
   InstanceGblSecDecrLong (&AccountingPtr->ThrottleCurrentlyProcessing);

   /* only call this function the one time (e.g. internal redirects) */
   rqptr->rqPathSet.ThrottleBusy =
      rqptr->rqPathSet.ThrottleFrom =
      rqptr->rqPathSet.ThrottleIndex =
      rqptr->rqPathSet.ThrottleResume =
      rqptr->rqPathSet.ThrottleTo =
      rqptr->rqPathSet.ThrottleTimeoutBusy =
      rqptr->rqPathSet.ThrottleTimeoutQueue = 0;

   /* if there are no requests waiting to be made active */
   if (!tsptr->CurrentlyQueuedCount) return;

   /* note the REUSE of 'rqptr'! (having just zeroed the previous 'rqptr') */
   rqptr = (REQUEST_STRUCT*)(tsptr->QueuedList.HeadPtr->DataPtr);

   /* if no process elbow-room (should only be after sysadmin intervention) */
   if (rqptr->rqPathSet.ThrottleResume &&
       tsptr->CurrentlyProcessingCount >=
       (rqptr->rqPathSet.ThrottleResume -
        rqptr->rqPathSet.ThrottleTo) +
        rqptr->rqPathSet.ThrottleFrom) return;

   /* release the head of the queued requests (FIFO) */
   ThrottleRelease (rqptr, true);
}

/*****************************************************************************/
/*
The HTTPd supervisor has called this function on a throttle timeout.  If it's a
"process" timeout then, then check if there is an absolute limit on
concurrently processed requests, if not then just send it on it's way to the
previously buffered next function, otherwise it's a 503 "busy".  If it a "busy"
timeout generate a 503 "busy" and the send it to request run-down! 
*/ 

ThrottleTimeout (REQUEST_STRUCT *rqptr)

{
   LIST_ENTRY  *eptr;
   THROTTLE_STRUCT  *tsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCHING(rqptr) && (WATCH_MODULE(WATCH_MOD_THROTTLE)))
      WatchThis (rqptr, FI_LI, WATCH_MOD_THROTTLE, "ThrottleTimeout()");

   tsptr = &ThrottleArray[rqptr->rqPathSet.ThrottleIndex];

   if (WATCHING(rqptr) && WATCH_CATEGORY(WATCH_REQUEST))
      WatchThis (rqptr, FI_LI, WATCH_REQUEST, "THROTTLE timeout !AZ",
                 rqptr->rqPathSet.ThrottleTimeoutQueue ? "QUEUE" : "BUSY");

   if (rqptr->rqPathSet.ThrottleTimeoutQueue)
   {
      /**********************/
      /* timeout to process */
      /**********************/

      tsptr->TotalTimeoutQueueCount++;

      /* if to be processed but limited by absolute maximum on processing */
      if (rqptr->rqPathSet.ThrottleResume &&
          tsptr->CurrentlyProcessingCount >=
          (rqptr->rqPathSet.ThrottleResume -
           rqptr->rqPathSet.ThrottleTo) +
           rqptr->rqPathSet.ThrottleFrom)
      {
         /* can't begin processing, just sit and wait on the queue */
         if (rqptr->rqPathSet.ThrottleTimeoutBusy)
         {
            /* this time do not use the process timeout */
            rqptr->rqPathSet.ThrottleTimeoutQueue = 0;
            HttpdTimerSet (rqptr, TIMER_THROTTLE, 0);
         }
         else
            HttpdTimerSet (rqptr, TIMER_OUTPUT, 0);

         return;
      }

      /* remove the entry from the queue to begin processing */
      ThrottleRelease (rqptr, true);

      return;
   }

   /*******************/
   /* timeout to busy */
   /*******************/

   tsptr->TotalTimeoutBusyCount++;

   /* remove the entry from the queue to be terminated with 503 "busy" */
   ThrottleRelease (rqptr, false);
}

/*****************************************************************************/
/*
Remove the request specified by 'rqptr' from the appropriate throttled path's
queue.  If 'ToProcess' declare an AST to recommence it's processing.  If to
be taken off the queue and discarded generate a 503 "busy" status explicitly
call RequestEnd() as an AST.  By the time this function is called it has been
decided whether to process or terminate the particular request.
*/ 

ThrottleRelease
(
REQUEST_STRUCT *rqptr,
BOOL ToProcess
)
{
   THROTTLE_STRUCT  *tsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCHING(rqptr) && (WATCH_MODULE(WATCH_MOD_THROTTLE)))
      WatchThis (rqptr, FI_LI, WATCH_MOD_THROTTLE, "ThrottleRelease()");

   tsptr = &ThrottleArray[rqptr->rqPathSet.ThrottleIndex];

   if (WATCHING(rqptr) && WATCH_CATEGORY(WATCH_REQUEST))
      WatchThis (rqptr, FI_LI, WATCH_REQUEST,
"THROTTLE set:!UL,!UL,!UL,!UL current:!UL,!UL !AZ",
                 rqptr->rqPathSet.ThrottleFrom,
                 rqptr->rqPathSet.ThrottleTo,
                 rqptr->rqPathSet.ThrottleResume,
                 rqptr->rqPathSet.ThrottleBusy,
                 tsptr->CurrentlyProcessingCount,
                 tsptr->CurrentlyQueuedCount,
                 ToProcess ? "PROCESS" : "BUSY");

   ListRemove (&tsptr->QueuedList, &rqptr->ThrottleListEntry);

   /* careful, these counters can get scrambled during a mapping reload */
   if (tsptr->CurrentlyQueuedCount) tsptr->CurrentlyQueuedCount--;
   InstanceGblSecDecrLong (&AccountingPtr->ThrottleCurrentlyQueued);

   if (ToProcess)
   {
      /***********/
      /* process */
      /***********/

      /* declare an AST for the next function to be performed */
      SysDclAst (&RequestExecutePostThrottle, rqptr);

      tsptr->TotalFiFoCount++;
      tsptr->CurrentlyProcessingCount++;
      if (tsptr->CurrentlyProcessingCount > tsptr->MaxProcessingCount)
         tsptr->MaxProcessingCount = tsptr->CurrentlyProcessingCount;
      InstanceGblSecIncrLong (&AccountingPtr->ThrottleCurrentlyProcessing);
   }
   else
   {
      /********/
      /* busy */
      /********/

      /* remove this request's claim to fame */
      rqptr->rqPathSet.ThrottleBusy =
         rqptr->rqPathSet.ThrottleFrom =
         rqptr->rqPathSet.ThrottleIndex =
         rqptr->rqPathSet.ThrottleResume =
         rqptr->rqPathSet.ThrottleTo =
         rqptr->rqPathSet.ThrottleTimeoutQueue =
         rqptr->rqPathSet.ThrottleTimeoutBusy = 0;

      rqptr->rqResponse.HttpStatus = 503;
      ErrorGeneral (rqptr, MsgFor(rqptr,MSG_GENERAL_TOO_BUSY), FI_LI);

      /* declare an AST to run-down the request */
      SysDclAst (&RequestEnd, rqptr);
   }

   /* reinitialize the timer for output */
   HttpdTimerSet (rqptr, TIMER_OUTPUT, 0);

   /* indicate it's no longer queued */
   rqptr->ThrottleListEntry.DataPtr = NULL;
}

/*****************************************************************************/
/*
Scan through all throttle structures looking for those with queued requests. 
Either release all the queued requests for processing, or just release the one
with the specified connect number.  This "release" is completely unconditional.
That is a non-extreme-prejudice release sets requests processing regardless of
any processing limitations in the throttle rules!!  Return the number of
dequeued requests.
*/ 

int ThrottleControl
(
BOOL WithExtremePrejudice,
int ConnectNumber
)
{
   int  idx, status,
        DequeuedCount;
   LIST_ENTRY  *leptr;
   REQUEST_STRUCT  *rqptr;
   THROTTLE_STRUCT  *tsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCH_MODULE(WATCH_MOD_THROTTLE))
      WatchThis (NULL, FI_LI, WATCH_MOD_THROTTLE,
                 "ThrottleControl() !UL", WithExtremePrejudice);

   DequeuedCount = 0;
   for (idx = 0; idx < MappingMetaPtr->ThrottleTotal; idx++)
   {
      tsptr = &ThrottleArray[idx];
      if (!tsptr->CurrentlyQueuedCount) continue;

      /* scan through all entries on this list */
      leptr = tsptr->QueuedList.HeadPtr;
      while (leptr)
      {
         rqptr = (REQUEST_STRUCT*)leptr->DataPtr;
         /* IMMEDIATELY get a pointer to the next in the list */
         leptr = leptr->NextPtr;

         /* if we're looking for a particular request and this is not it */
         if (ConnectNumber && rqptr->ConnectNumber != ConnectNumber) continue;

         if (WithExtremePrejudice)
            ThrottleRelease (rqptr, false);
         else
            ThrottleRelease (rqptr, true);

         DequeuedCount++;

         /* if dequeuing just the one request then return now */
         if (ConnectNumber) return (DequeuedCount);
      }
   }

   if (WATCH_CAT && Watch.Category)
      WatchThis (NULL, FI_LI, WATCH_REQUEST, "THROTTLE control !AZed !UL",
                 WithExtremePrejudice ? "terminat" : "releas", DequeuedCount);

   return (DequeuedCount);
}

/*****************************************************************************/
/*
Provide a report on the current state and history of any throttled paths.
*/ 

ThrottleReport
(
REQUEST_STRUCT *rqptr,
REQUEST_AST NextTaskFunction
)
{
   static char  BeginPage [] =
"<P><TABLE CELLPADDING=1 CELLSPACING=0 BORDER=0>\n\
<TR><TH COLSPAN=6></TH>\
<TH COLSPAN=6 VALIGN=bottom><U>Queue</U></TH>\
<TH>&nbsp;<TH>\
<TH COLSPAN=2 VALIGN=bottom><U>Processing</U></TH>\
</TR>\
<TR>\
<TH></TH>\
<TH><U>Path</U>&nbsp;&nbsp;</TH>\
<TH><U>Total</U>&nbsp;&nbsp;</TH>\
<TH><U>Busy</U>&nbsp;&nbsp;</TH>\
<TH>&nbsp;<TH>\
<TH><U>Total</U>&nbsp;&nbsp;</TH>\
<TH><U>Cur</U>&nbsp;&nbsp;</TH>\
<TH><U>Max</U>&nbsp;&nbsp;</TH>\
<TH><U>FIFO</U>&nbsp;&nbsp;</TH>\
<TH><U>T/Oq</U>&nbsp;&nbsp;</TH>\
<TH><U>T/Ob</U>&nbsp;&nbsp;</TH>\
<TH>&nbsp;<TH>\
<TH><U>Cur</U>&nbsp;&nbsp;</TH>\
<TH><U>Max</U></TH>\
<TH>&nbsp;&nbsp;</TH>\
</TR>\n\
<TR HEIGHT=5></TR>\n";

   static char  ThrottlePathFao [] =
"<TR><TH ALIGN=right>!3ZL&nbsp;&nbsp;</TH>\
<TD ALIGN=left>!&;AZ&nbsp;&nbsp;throttle=!UL,!UL,!UL,!UL,\
!2ZL:!2ZL:!2ZL,!2ZL:!2ZL:!2ZL&nbsp;&nbsp;</TD>\
<TD ALIGN=right>!UL&nbsp;&nbsp;</TD>\
<TD ALIGN=right>!UL&nbsp;&nbsp;</TD>\
<TD>&nbsp;<TD>\
<TD ALIGN=right>!UL&nbsp;&nbsp;</TD>\
<TD ALIGN=right>!UL&nbsp;&nbsp;</TD>\
<TD ALIGN=right>!UL&nbsp;&nbsp;</TD>\
<TD ALIGN=right>!UL&nbsp;&nbsp;</TD>\
<TD ALIGN=right>!UL&nbsp;&nbsp;</TD>\
<TD ALIGN=right>!UL&nbsp;&nbsp;</TD>\
<TD>&nbsp;<TD>\
<TD ALIGN=right>!UL&nbsp;&nbsp;</TD>\
<TD ALIGN=right VALIGN=top>!UL&nbsp;&nbsp;</TD>\
</TR>\n\
<TR>\
<TH></TH>\
<TD BGCOLOR=\"#eeeeee\"></TD>\
<TD ALIGN=right BGCOLOR=\"#eeeeee\">!UL%&nbsp;&nbsp;</TD>\
<TD ALIGN=right BGCOLOR=\"#eeeeee\">!UL%&nbsp;&nbsp;</TD>\
<TD COLSPAN=5 BGCOLOR=\"#eeeeee\"></TD>\
<TD ALIGN=right BGCOLOR=\"#eeeeee\">!UL%&nbsp;&nbsp;</TD>\
<TD ALIGN=right BGCOLOR=\"#eeeeee\">!UL%&nbsp;&nbsp;</TD>\
<TD ALIGN=right BGCOLOR=\"#eeeeee\">!UL%&nbsp;&nbsp;</TD>\
<TD COLSPAN=4 BGCOLOR=\"#eeeeee\"></TD>\
</TR>\n";

   static char  ButtonsFao [] =
"</TABLE>\n\
<P><I><SUP>*</SUP>throttle=n1,n2,n3,n4,to1,to2\n\
<FONT SIZE=-1>\n\
<BR>&nbsp;&nbsp;n1, concurrent requests before queuing\n\
<BR>&nbsp;&nbsp;n2, concurrent requests before FIFO processing\n\
<BR>&nbsp;&nbsp;n3, concurrent requests before FIFO processing ceases again\n\
<BR>&nbsp;&nbsp;n4, concurrent requests before immediate &quot;busy&quot;\n\
<BR>&nbsp;&nbsp;to1, maximum period queued before processing \
(if not limited by n3)\n\
<BR>&nbsp;&nbsp;to2, maximum period queued before &quot;busy&quot; \
(from expiry of any to1)\n\
</FONT></I>\n\
<BR><I><SUP>**</SUP>all percentages are of path total</I>\n\
<HR SIZE=1 NOSHADE WIDTH=60% ALIGN=left>\n\
<TABLE CELLPADDING=5 CELLSPACING=0 BORDER=0>\n\
<TR><TD>\n\
<FORM METHOD=GET ACTION=\"!AZ\">\n\
<INPUT TYPE=submit VALUE=\" Zero \">\n\
</FORM>\n\
</TD><TD>&nbsp;</TD><TD>\n\
<FORM METHOD=GET ACTION=\"!AZ\">\n\
<INPUT TYPE=submit VALUE=\" Release \">\n\
</FORM>\n\
</TD><TD>&nbsp;</TD><TD>\n\
<FORM METHOD=GET ACTION=\"!AZ\">\n\
<INPUT TYPE=submit VALUE=\" Terminate \">\n\
</FORM>\n\
</TD></TR>\n\
</TABLE>\n\
</BODY>\n\
</HTML>\n";

   int  idx, status,
        Percent503,
        PercentFiFo,
        PercentQueue,
        PercentTimeoutBusy,
        PercentTimeoutQueue;
   unsigned long  *vecptr;
   unsigned long  FaoVector [32];
   double  fScratch,
           fTotalCount;
   THROTTLE_STRUCT  *tsptr;
   MAP_RULE_META  EmptyRuleJustInCase;
   MAP_RULE_META  *mrptr;

   /*********/
   /* begin */                                    	
   /*********/

   if (WATCHING(rqptr) && (WATCH_MODULE(WATCH_MOD_THROTTLE)))
      WatchThis (rqptr, FI_LI, WATCH_MOD_THROTTLE,
                 "ThrottleReport() !&A !UL",
                 NextTaskFunction, MappingMetaPtr->ThrottleTotal);

   rqptr->rqResponse.PreExpired = PRE_EXPIRE_ADMIN;
   RESPONSE_HEADER_200_HTML (rqptr);
   AdminPageTitle (rqptr, "Throttle Report", BeginPage);

   for (idx = 0; idx < MappingMetaPtr->ThrottleTotal; idx++)
   {
      tsptr = &ThrottleArray[idx];
      if (tsptr->TotalCount)
      {
         fTotalCount = (double)tsptr->TotalCount;

         fScratch = (double)tsptr->Total503Count * 100.0 / fTotalCount;
         Percent503 = (int)fScratch;
         if (modf (fScratch, &fScratch) >= 0.5) Percent503++;

         fScratch = (double)tsptr->TotalFiFoCount * 100.0 / fTotalCount;
         PercentFiFo = (int)fScratch;
         if (modf (fScratch, &fScratch) >= 0.5) PercentFiFo++;

         fScratch = (double)tsptr->TotalQueuedCount * 100.0 / fTotalCount;
         PercentQueue = (int)fScratch;
         if (modf (fScratch, &fScratch) >= 0.5) PercentQueue++;

         fScratch = (double)tsptr->TotalTimeoutBusyCount * 100.0 / fTotalCount;
         PercentTimeoutBusy = (int)fScratch;
         if (modf (fScratch, &fScratch) >= 0.5) PercentTimeoutBusy++;

         fScratch = (double)tsptr->TotalTimeoutQueueCount * 100.0 / fTotalCount;
         PercentTimeoutQueue = (int)fScratch;
         if (modf (fScratch, &fScratch) >= 0.5) PercentTimeoutQueue++;
      }
      else
         Percent503 = PercentFiFo = PercentQueue =
         PercentTimeoutBusy = PercentTimeoutQueue = 0;

      /* get details of the throttle rule using the index number */
      if (!(mrptr = MapUrl_ThrottleRule (idx)))
      {
         memset (mrptr = &EmptyRuleJustInCase, 0, sizeof(MAP_RULE_META));
         EmptyRuleJustInCase.TemplatePtr = "?";
      }

      vecptr = FaoVector;
      *vecptr++ = idx + 1;
      *vecptr++ = mrptr->TemplatePtr;
      *vecptr++ = mrptr->mpPathSet.ThrottleFrom;
      *vecptr++ = mrptr->mpPathSet.ThrottleTo;
      *vecptr++ = mrptr->mpPathSet.ThrottleResume;
      *vecptr++ = mrptr->mpPathSet.ThrottleBusy;
      *vecptr++ = mrptr->mpPathSet.ThrottleTimeoutQueue / 3600;
      *vecptr++ = (mrptr->mpPathSet.ThrottleTimeoutQueue % 3600) / 60;
      *vecptr++ = (mrptr->mpPathSet.ThrottleTimeoutQueue % 3600) % 60;
      *vecptr++ = mrptr->mpPathSet.ThrottleTimeoutBusy / 3600;
      *vecptr++ = (mrptr->mpPathSet.ThrottleTimeoutBusy % 3600) / 60;
      *vecptr++ = (mrptr->mpPathSet.ThrottleTimeoutBusy % 3600) % 60;
      *vecptr++ = tsptr->TotalCount;
      *vecptr++ = tsptr->Total503Count;
      *vecptr++ = tsptr->TotalQueuedCount;
      *vecptr++ = tsptr->CurrentlyQueuedCount;
      *vecptr++ = tsptr->MaxQueuedCount;
      *vecptr++ = tsptr->TotalFiFoCount;
      *vecptr++ = tsptr->TotalTimeoutQueueCount;
      *vecptr++ = tsptr->TotalTimeoutBusyCount;
      *vecptr++ = tsptr->CurrentlyProcessingCount;
      *vecptr++ = tsptr->MaxProcessingCount;
      *vecptr++ = Percent503;
      *vecptr++ = PercentQueue;
      *vecptr++ = PercentFiFo;
      *vecptr++ = PercentTimeoutQueue;
      *vecptr++ = PercentTimeoutBusy;

      status = NetWriteFaol (rqptr, ThrottlePathFao, &FaoVector);
      if (VMSnok (status)) ErrorNoticed (status, "NetWriteFaol()", FI_LI);
   }

   if (!idx)
      NetWriteFao (rqptr,
"<TR><TD COLSPAN=12 ALIGN=center><I>(none)</I></TD></TR>\n",
                   NULL);

   vecptr = FaoVector;
   *vecptr++ = ADMIN_CONTROL_THROTTLE_ZERO;
   *vecptr++ = ADMIN_CONTROL_THROTTLE_RELEASE;
   *vecptr++ = ADMIN_CONTROL_THROTTLE_TERMINATE;

   status = NetWriteFaol (rqptr, ButtonsFao, &FaoVector);
   if (VMSnok (status)) ErrorNoticed (status, "NetWriteFaol()", FI_LI);

   SysDclAst (NextTaskFunction, rqptr);
}

/*****************************************************************************/

