University of Washington, Bothell

CSS 485: Introduction to Artificial Neural Networks
Fall 2002
Homework Assignment 3: Dynamics and Optimization

Assigned: Wednesday, October 30, 2002
Due: Wednesday, November 6, 2002 (start of class)

1. Do exercise 5.3(b), using MATLAB (or some other plotting software of your choice, as long as it will do a reasonable job) to produce the plot.

2. Write a program in the programming language of your choice to find the minimum of a function using gradient descent. The function will be written (as a function) in the same programming language, too, and compiled and linked in if you are using a compiled language (in other words, you don't need to input and parse algebraic expressions). Your program should output its progress during execution, with columns being: iteration number, $x_1$, $x_2$, ..., $x_n$, $f(x_1, \ldots, x_n)$. Include a printout of your program in your report.

   Use your program to find the minima of the following functions:

   • $f(\mathbf{x}) = x_1^2 - x_1 x_2 + 3x_2^2$
   • $f(\mathbf{x}) = 5x_1^2 - 6x_1 x_2 + 5x_2^2 + 4x_1 + 4x_2$

   For each, experiment with $\eta$ to come up with a value that will cause reasonably rapid convergence while still being stable. In your report, for each function include:

   1. The value of $\eta$ you used.
   2. A contour plot of the function, with trajectory points superimposed (points connected by lines showing the trajectory) for a representative run of your program.
   3. An answer to questions, ``Does the choice of initial search location affect the minimum found? Why or why not?''

3. Implement the linear neural network error function (objective function) found on p. 118 of the textbook for use with your program above. Use it and your program to find the best weights for the following input/output pairs (training set) for a two-input, one-output neural network:

   $$\left\{ (\mathbf{x}_1 = \left[\begin{array}{r}2\\ 1\end{array}\ri... ...thbf{x}_2 = \left[\begin{array}{r}-1\\ 1\end{array}\right], y_2^p = 0) \right\}$$