Project Overview

I have a short powerpoint presentation which gives the overall architecture and thread states for my implementation, see wuthreads.ppt.

Finally, I encourage each of you to implement your project iteratively. For example, first make sure you know how to use sigsetjump and siglongjmp. Then try saving several contexts and altering the contents of the sigjump_buf by changing the stack and PC pointers. From there you can try creating threads and adding them to a simple FIFO queue, and so forth. I have created three simple programs to illustrate what I mean, see setjmp1.c, setjmp2.c and setjmp3.c. You can compile these programs using the command

  gcc -o setjmp1 setjmp1.c
  gcc -o setjmp2 setjmp2.c
  gcc -o setjmp3 -I$WUSRC setjmp3.c
  # where $WUSRC references the directory where you have located the wulib
  # sources (needed for the include file wulib/queue.h).

Project Requirements

Phase 1

• Periodic clock handler: used for implementing timeout queues (aka callout queue), thread CPU usage accounting, time-slicing and periodic scheduling decisions. The clock handler drives the preemptive, round-robin scheduling discipline. It must use the system's virtual interval timer, see setitimer(2), so that time increments only when the process is running, we will call this our virtual time or simply time. For example, the following enables this timer so that a SIGVTALRM signal is sent to the process every T seconds (you must define the SIGVTALRM handler clock_handler yourselves):

    // install the signal handler before starting the timer
    struct sigaction sa;
    struct itimerval tout;
    // tick2usec() is defined in <wulib/timer.h> and
    // the constant WTH_CLOCK_TICKS indicates the number of local OS ticks
    // correspond to one of our virtual ticks (i.e. our period). For
    // example if our virtual tick size is 20msec and the local tick size
    // is 10ms then WTH_CLOCK_TICKS = 2.
    long T = tick2usec(WTH_CLOCK_TICKS);
  
    /* setup the signal action */
    if (sigemptyset(&sa.sa_mask) != 0) {
      wulog(wulogClock, wulogFatal, "start_clock: error initializing sa_mask.\n");
      return -1;
    }
  
    // bind our periodic clock handler to the SIGVTALRM Signal
    sa.sa_flags   = 0;
    sa.sa_handler = clock_handler;
    if (sigaction(SIGVTALRM, &sa, NULL) != 0) {
      debugLog(wulogClock, wulogFatal, "start_clock: error calling sigaction\n", errno);
      return -1;
    }
  
    
    wulog(wulogClock, wulogInfo, "Setting T to %d local ticks or %d usec\n",
          WTH_CLOCK_TICKS, T);
  
    /* Period interval, used to rest counter */
    tout.it_interval.tv_sec  = 0;
    tout.it_interval.tv_usec = T; /* periodic interval, in usecs */
    /* First (immediate) timeout interval */
    tout.it_value.tv_sec     = 0;
    tout.it_value.tv_usec    = T; /* time to next signal */
  
    /* 
    * setitimer(2) takes three values:
    *   1) int which - there are three different timers that can be used by
    *              by a process ITIMER_REAL, ITIMER_VIRTUAL and ITIMER_PROF.
    *         ITIMER_VIRTUAL - decrements only when the process is running and
    *              delivers SIGVTARLAM when it reaches zero. We will use this timer.
    *         ITIMER_REAL - decrements with "real" time and sends SIGARLM when
    *              reaches zero. We will not use this timer.
    *         ITIMER_PROF - decremented with either the process is running (user
    *              mode) or the system is running on its behalf (kernel mode, process
    *              context). We will not use this timer.  Sends SIGPROF when expires.
    *   2) struct itimerval *value - sets indicated timer to timer value.
    *   3) struct itimerval *ovalue - the old value (we just set this to NULL).
    *
    * value->it_interval is set the to desired timeout interval. The clock will count
    * down from this value, send the indicated signal, then start all over again.
    *
    * Set the period (T) to a multiple of the system clock interval, you can use
    * the functions defined in wulib/timer.h
    * */
    if (setitimer(ITIMER_VIRTUAL, &tout, NULL) < 0) {
      log_msg(WLOG_MOD_CLOCK | LOG_FATAL, "start_clock: Error setting interval timer");
      return -1;
    }
  

• Preemptive, priority-driven, hierarchical scheduler: Your top-level scheduler implicitly or explicitly assigns a range of global priorities to each loaded scheduler. The effect is to have an ordered list of scheduling classes from high to low priority. Then each time the scheduler runs it selects the highest priority runnable thread that has been released by its associated scheduling class. If a ready thread has a higher priority than the currently running thread, then the scheduler framework preempts the current thread, hands it back to its scheduling class and performs a context switch to the selected thread.

  Context switches can be realized using the sigsetjmp(3) and siglongjmp(3) library functions. sigsetjmp saves the current register values and blocked signals in the buffer passed to the library call. siglongjmp then restores the register values and blocked signal mask that are stored in the buffer. By performing a little surgery on the sigjmp_buf we can change the program counter (PC) and stack pointer (%esp) values to ones of our choosing, that is how we implement user-level threads. For more information on non-local exists see the glibc documentation. When creating a new thread first allocate memory for the stack using malloc. Then execute the sigsetjmp and exchange the saved stack pointer value for the just allocated memory (recall that stacks grow down) and set the program pointer so that the thread's entry function is called. I outline the steps below (see supplied header files for description of types and default constants):

    // declare this as volatile so we are sure to get the correct
    // value on return from a non-local goto (setjmp). See the glibc docs.
    volatile int stacksz; 
    volatile uint32_t *hwstate;
  
    // allocate space to store the threads state
    wth_t *wth = alloc_wth(...);
    stacksz = user supplied or WTH_DEFSTACK_SIZE;
  
    wth->core.entry = entry; // entry function
    wth->core.arg   = arg;   // arguments to entry function
    wth->core.stack = calloc(1, stacksz);
  
    if (sigsetjmp(wth->core.hwstate, 1) != 0) {
      ... errror ...
      return -1;
    }
  
  

  Next we need to doctor this state so we can execute with the new stack and the user supplied entry function. Rather then directly calling the user entry funtion, entry, I use a wrapper function called wth_start. I do this so that I can perform any initialization/cleanup operations before I call/after I return from the user supplied function.

    STATIC void wth_start(void);
    void wth_start(void)
    {
      ...
      call the entry function for the currently running thread
      ...
    }
  
  

  Now we can alter the saved state,

    hwstate        = (uint32_t *)wth->core.hwstate->__jmpbuf;
    hwstate[JB_SP] = (uint32_t)((char *)wth->core.stack + stacksz);
    hwstate[JB_PC] = (uint32_t)wth_start;
  
  

  Then context switches are as simple as the following ... well, almost this simple:

    if (sigsetjmp(old->core.hwstate, 1) == 0)
      siglongjmp(new->core.hwstate, 1);
  

• Implement FIFO and Round Robin scheduling classes: FIFO and Round Robin scheduling policies: All policies must support preemption and class specific priorities as outlined above. A user must be able to select relative priorities within each class, for example each class should at least support priorities from 0 to 15. In addition the round robin class associates a time quantum (i.e. time-slice) with each thread. It is acceptable for the time quantum to be statically configured at compile time or to dynamically change at run time based on a threads relative use of system resources.

Phase 2

• The hierarchal scheduling framework must support four different policies: realtime, interactive, besteffort (background) and system background. The framework implements a simple priority-driven, preemptive scheduling algorithm with the realtime policy having the highest priority and the system background policy the lowest. You should have implemented this framework in the first phase of this design (Project2).

• The library must support and provide interfaces for a blocking lock (mutex) and a condition variable (signal-and-continue semantics). You must use the include file sync.h in your implementation. In this file I include another header file called wuthreads/wthtypes.h to define the data struct waitq_t. You must remove this include (wuthreads/wthtypes.h) and replace with your own definition of waitq_t.

• You must add a functoin to your wth.h interface that implements a timed sleep operations. The function wth_twait(unsigned int msec) causes the calling thread to sleep for the indicated number of msecs, after whic hthe caller is woken and again eligable for scheduling.

  How operating systems implement timeout queues

  Operating sytem clock interrupt handlers are generally responsible for asserting timeout events requested by users and other operating system modules (for example, wake thread in 200msecs). The traditional approach is to have a sorted list of operations called a callout queue (also known as a timer queue). A callout (an element of the callout queue) includes both an expiration time and a kernel specified function to be executed. For example, the following function can be used to register a callout:

  int to_ID = timeout (void (*fn)(), caddr_t arg, long delta);
  

  where fn is the function to invoke, arg is the argument to pass to fn and delta is the time interval in CPU ticks. These parameters are recorded in a callout object. In BSD based systems the queue (actually implemented as a linked list of callouts) is sorted by time to expire. Each entry stores the difference between its time to fire and that of the previous callout. The kernel decrements the time of the first entry at each clock tick and issues the callout when the time reaches zero.

  An alternative approach (and one I recommend) uses a similarly sorted list but stores the absolute time of expiration for each entry. This way, at each clock tick the current absolute time (in clock ticks) is compared to the first element of the list. When they are equal the callout is fired.

  Both of these schemes optimize the lookup time (i.e. constant lookup time) at the expense of the insertion time. The goal is to minimize the time required to process the callout queue which is done every clock tick. An alternative approach attempts to address the average insertaion time by using a timing wheel. A timing wheel is a fixed-size circular array of callout queues. At each clock tick, the interrupt handler advances a current time pointer to the next element in the array, wrapping around at the end of the array. If there are any callouts on the queue their expiration time is checked. New callouts are inserted on the queue that is N elements away from the current queue where N is the time to fire measured in clock ticks. In other words, the callouts are hashed based on their expiration time. Each callout queue may or may not be sorted.

Phase 3

• For this phase we focus on the hierarchal scheduling frameworks three top-priority scheduling policies: realtime, interactive and besteffort. The framework implements a simple priority-driven scheduling algorithm with the realtime policy having the highest priority and the besteffort policy the lowest. I am using generic terms on purpose, but you may find it convenient to think of realtime as periodic, interactive as sporadic and besteffort as aperiodic.
• You must implement two new scheduling classes: earliest deadline first (EDF) and rate monotonic (RM). Any of the four different scheduling classes (EDF, RM, FIFO or RR) can be assigned to any of the top-level policies (real-time, interactive or besteffort).
• One problem that we must deal with is initializing our environment where we are creating real-time threads with priorities higher than the main thread's. To deal with this I want you to add a method to the scheduler's interface called sched_start. The main thread will still call wth_libinit which initializes the library (including the scheduler) and creates the main thread but it does not "start" the scheduler. In other words, the scheduler with allow an application to create threads, add them to scheduling classes but it will not attempt to preempt the main thread nor make any scheduling decision until after the sched_start method is called. This permits the main thread to create all the real-time threads, initialize any data structures and generally "get out of the way" before the higher priority, real-time threads start to run.

Designing and implementing the library is half the story, you must also create a test environment for validating your library. Specifically you must create tests that verify the operation of your synchronization mechanisms (mutex and condition variables), the scheduling framework and new scheduling classes.

• Verify the synchronization primitives: demonstrate that threads will indeed block when a lock is head and woken when it is released. Likewise, verify condition variables and both the signal and broadcast operation.
• For the two new real-time scheduling classes demonstrate that they achieve, or come close to, the theoretical schedulable utilization (minus overhead in your library). Your tests should test with low utilization, those close to the theoretical maximum and utilizations greater than 1 (i.e. overload operation). A report must be printed out at the completion of the test identifying the percent of jobs that missed their deadlines, the target utilization and actual utilization.
• Verify operation when the realtime and interactive policies are both instantiated with EDF, RM and a combination of the two (threads assigned to each policy). For this test I want to know the percent of missed deadlines within each policy during overload and non-overload conditions.

Using wulib

There are several utility functions and macros available in the wulib library. You can use the logging facility (log.{h,c}), timer functions (timer.{h,c}) and queue manipulation code (queue.h and pqueue.{h,c}).

• logging: you are to include wlog.{h,c} in the cse422 wth directory. This module defines addition logging modules that I will use when testing your code and can benefit your debugging and verification efforts. See the test application code for initializing the logging facility (log_init()) and printing log messages using log_msg(<Module | Level>, "format", args);
• Queue manipulation: In class we have reviewed the basic queue management macros defined in queue.c (see C.ppt). I suggest using this header file for all you queue management functions, see the qtest.c file in the wulib directory. I have also provided a priority queue implementation defined in pqueue.{h,c} but for the initial implementation I suggest no using this as the queue.h macros are more efficient.
• Timer functions: in timer.h I have defined several utility functions that you may find useful. In particular tick2msec() (and friends) provide a simple mechanism for determine the platform's tick (clock) interval at run time.

Required interface specification

Phase 1

You must implement the following interface, the interface is defined in wth.h which you must use without modification (read the header file for a more detailed description of the API):

• int  wth_libinit(wth_attr_t *attr, int *sched_policies); initialize library.
• void wth_clock(void); This is called by your clock handler, it performs any required accounting functions and calls the scheduler to ensure the highest priority thread is running.
• void wth_exit(int status); An application thread calls this to terminate. The status value is passed to the "parent" thread when it does the corresponding join operation.
• int wth_create (wth_attr_t *attr, void (*entry)(void *), void *arg); Creates a new thread assigning it to the requested scheduling class. See the data type definitions. The function "entry" is called with "arg" as its only argument.
• int wth_this(void); Returns this thread's ID, an integer.
• void wth_yield(void); Yield CPU to a runnable thread with an equal priority.
• int wth_join(int tid, int *result); Suspend the calling thread until the thread with ID tid terminates. The argument results is set to equal the value of the status parameter passed to the wth_exit() method.

Phase 2

You must implement the following interface, the interface is defined in sync.h which you must use with the modification specified above, for a more detailed description of the API see the header file sync.h (mutex and condition variable) or wth.h (timed wait):

• Mutual exclusion lock (sync.h)

  int wth_lock_init(wth_lock_t lock, wth_lock_attr_t attr);

  Create a new mutual exclusion lock with the attributes specified in the wth_lock_attr_t object. Currently there is only one attribute define: recursive. If WTH_MUTEX_ATTR_RECURSIVE is specified then recursive calls to wth_lock will not deadlock. If WTH_MUTEX_ATTR_DEFAULT is specified then recursive calls to wth_lock will result in a deadlock. The arguments lock and attr must be allocated by the calling thread and will be initialized by this method. Returns -1 on error and 0 on success.

  int wth_lock_destroy(wth_lock_t lock);

  Destroys the lock object passed as an argument (it can not longer be used). Returns -1 on error and 0 on success.

  int wth_lock(wth_lock_t lock);

  Must first initialize the lock object by calling wth_lock_init. Then the method wth_lock implements a mutual exclusion lock using the object lock. If the lock is currently held by another thread the calling thread is put to sleep until the lock is released. Returns -1 on error and 0 on success.

  int wth_unlock(wth_lock_t lock);

  The caller must currently own the lock. Calling wth_unlock releases the lock and if any threads are waiting for the lock exactly one thread will be woken. Returns -1 on error and 0 on success.

• Condition variable (signal-and-continue, sync.h)

  int wth_condvar_init(wth_condvar_t cv, wth_condvar_attr_t attr);

  Condition variables cv and attributes attr must be allocated by the calling thread, this method will then initialize the objects. Currently no attributes are defined for condition variables. Returns -1 on error and 0 on success.

  int wth_condvar_destroy(wth_condvar_t);

  Frees any allocated resources, must call this method when you are done using a condition variable. Returns -1 on error and 0 on success.

  int wth_cvwait(wth_condvar_t cv, wth_lock_t lock);

  Implements the wait method for a condition variable. The lock and cv objects must have been initialized before calling this method. The calling thread will go to sleep utile wth_cvsig or wth_cvbcast is called to wake it up. Returns -1 on error and 0 on success.

  int wth_cvsig(wth_condvar_t cv);

  Implements the signal operation on a condition variable. Exactly one thread (the longest waiting) is woken. Returns -1 on error and 0 on success.

  int wth_cvbcast(wth_condvar_t cv);

  Wakes all sleeping threads on a condition variable. Returns -1 on error and 0 on success.

• Timed wait (wth.h)

  int wth_twait(unsigned int msec);

  Puts caller to sleep for at least msec milli seconds. The parameter msec is rounded up to a multiple of the libraries clock interval. For example if the tick size is 10msec and if the requested sleep time is .105 secods the nthe thread should sleep for .110 seconds (110 milli seconds). Returns 0 on success and -1 on error.

Phase 3

Platform Requirements

You must use Linux for the project.

Library Verification

I have added two test programs called callout.c and testapp.c: testapp.c is to verify the first phase of the project and callout.c is for the second. The following commands can be used to compile your test applications on the CEC Linux hosts, where $WUSRC references the class direcotry with the wulib sources and $WUCSE is your top-level project directory:

      gcc  -DMyOS=Linux -DPTHREADS -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -Wall -g -DWUDEBUG \
      -I${WUSRC} -I/usr/include -I${WUCSE} -I. -o Linux/callout.o -c callout.c

      gcc  -DMyOS=Linux -DPTHREADS -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -Wall -g -DWUDEBUG \
      -o Linux/callout Linux/callout.o [Your wuthreads library] -L${WUSRC}/wulib/Linux -lwu -lm

      gcc  -DMyOS=Linux -DPTHREADS -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -Wall -g -DWUDEBUG  \
      -I${WUSRC} -I/usr/include -I${WUCSE} -I.  -o Linux/testapp.o -c testapp.c

      gcc  -DMyOS=Linux -DPTHREADS -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -Wall -g -DWUDEBUG \
      -o Linux/testapp Linux/testapp.o [Your wuthreads library] \
      -o Linux/callout Linux/callout.o [Your wuthreads library] -L${WUSRC}/wulib/Linux -lwu -lm

      If you created an archive library of your wuthreads source then you can
      add the following where it says [Your wuthreads library]:

      -L[your path]/Linux -lwth

      Or list all the object files.
    

Turning in your project