OPTIMIZATION TECHNIQUES: MATH:4820, Section 0001

Cross-listed: CS:4720, Section 0001

Course meeting time and place: 10:30-11:20AM MWF, 105 MLH

Department of Mathematics

Course ICON site: To access the course site, log into Iowa Courses Online (ICON) using your Hawk ID and password.

Instructor:

• Laurent O. Jay
• Office location: 225M MLH
• Student drop-in hours: 2:30-3:20PM MWF. I am also available by appointment if you are unable to attend my drop-in hours. For students missing classes without a valid justification, drop-in hours are not meant to be a private lesson to repeat what was covered in class.
• Phone: (319) 335-0898
• E-mail: laurent-jay@uiowa.edu
• DEO: Prof. Ryan Kinser, 14 MLH, E-mail: ryan-kinser@uiowa.edu

Prerequisites:

• MATH:1560 Engineering Math II: Multivariable Calculus OR MATH:1860 Calculus II
• MATH:2550 Engineering Math III: Matrix Algebra OR MATH:2700 Introduction to Linear Algebra
• Some computer programming experience, preferably MATLAB, will be helpful.

Description of course:

1. Unconstrained optimization:
• Steepest-descent method
• Newton-like methods
• Quasi-Newton methods
• Linear/nonlinear conjugate gradient methods
• Interval reduction methods
• Line-search methods
• Trust-region methods
• Local and global convergence
2. Nonlinear equations:
• Newton's method
• Modified Newton's methods
• Broyden's (quasi-Newton) method
• Inexact Newton methods
• The bisection method
• Line-search methods and merit functions
• Trust-region methods
• Local and global convergence
3. Constrained optimization:
• Lagrange multipliers
• Karush-Kuhn-Tucker conditions
• Line-search methods and merit functions
• Active-set methods (for inequality constraints)
• Penalty function methods (for equality constraints)
• Reduced-gradient and gradient-projection methods
• Augmented Lagrangian and projected Lagrangian methods
• Barrier methods (for inequality constraints)
• Interior-point methods (for inequality constraints)
• Sequential linearly constrained programming
• Sequential quadratic programming
4. More topics in optimization:
• Convexity
• Linear programming and the simplex method
• Quadratic programming
• Duality
• Nonlinear least-squares problems
• Variational calculus
• Nonsmooth optimization
• Dynamic optimization and the maximum principle of Pontryagin
• Dynamic programming and the Hamilton-Jacobi-Bellman equation
• Neural networks and the backpropagation algorithm
• Stochastic optimization
• Simulated annealing
• Genetic algorithms

Learning Objectives: This course is at a senior undergraduate and beginning graduate level and it is assumed that you can work along the course in an independent fashion. This course will cover modern optimization techniques for both constrained and unconstrained optimization with continuous (as opposed to discrete) variables. Given its strong links to optimization techniques, the numerical solution of nonlinear equations will also be considered. At the end of the course the student should master some essential issues in numerical optimization.

Class procedures: The majority of each class period will be lecture oriented. It is strongly advised to read the material to be discussed before coming to class. Therefore, if there is a difficult point, you will know beforehand where it arises, so that you can benefit from the lecture more effectively. If a point remains unclear you can ask questions in class. Readings may be assigned. Standard out-of-class preparation is at least six hours per week.

Textbook/Materials:

• (CLZ) An Introduction to Optimization: With Applications to Machine Learning, by Edwin K. P. Chong, Wu-Sheng Lu, Stanislaw H. Zak, Wiley, October 10, 2023, 5th Edition, 672 pages, ISBN-10: 1119877636, ISBN-13: 978-1119877639, list price: $131.95. The book on amazon.com.
• (K1) Free electronic book on Iterative Methods for Linear and Nonlinear Equations by Tim Kelley. If the link does not work, go to Download Books from SIAM. This textbook on SIAM for purchase, Softcover, ISBN-10: 0-89871-352-8, ISBN-13: 978-0-898713-52-7, list price: $57.00, SIAM member price $39.90 (becoming a SIAM member is free for students!). The book on amazon.com.

Additional useful readings:

• (NW) Free electronic book on Numerical optimization, by Jorge Nocedal and Stephen Wright, Springer, New York, Springer Series in Operations Research and Financial Engineering, 2006, Second edition, 664 pages, ISBN-10: 0387303030, ISBN-13: 978-0387303031, list price: $79.95. Library reference: MATH QA402.5 .N62 1999. Electronic version. The book on amazon.com.
• (K2) Free electronic book on Iterative Methods for Optimization by Tim Kelley. If the link does not work, go to Download Books from SIAM. This textbook on SIAM for purchase, list price: $61.00, SIAM member price $42.70 (becoming a SIAM member is free for students!). The book on amazon.com.
• Nonlinear programming by Dimitri P. Bertsekas, Athena Scientific, Belmont, MA, 1995, ISBN 1-886529-14-0, (MATH T 57.8 .B477 1995).
• Numerical methods for unconstrained optimization and nonlinear equations by John E. Dennis and Robert B. Schnabel, Prentice Hall, Englewood Cliffs, NJ, 1988, reprinted by SIAM publications, 1993, ISBN 0-89871-364-1, (MATH QA402.5 .D44 1983).
• Practical methods of optimization by R. Fletcher, Second edition, John Wiley & Sons, Chichester, New-York, 1987, ISBN 0-471-91547-5, (ENGINEERING QA402.5 .F43 1987).
• Practical optimization by Philip E. Gill, Walter Murray and Margaret H. Wright, Academic Press, New York, NY, 1981, ISBN 0-12-283950-1 (ENGINEERING QA402.5 G54).
• Linear and nonlinear programming by David G. Luenberger, second edition, Addison-Wesley Publ. Comp., Reading, MA, 1984, ISBN 0-201-15794-2, (MATH T 57.7 .L8 1984).
• Numerical Optimization: Theoretical and Practical Aspects by J. Frederic Bonnans, Jean Charles Gilbert, Claude Lemarechal, Claudia A. Sagastizbal Springer Series: Universitext, 2006, Second edition, 490 pages, ISBN-10: 3-540-35445-X, ISBN-13: 978-3-540-35445-1.

Academic Honesty and Misconduct: All students in CLAS courses are expected to abide by the CLAS Code of Academic Honesty. Undergraduate academic misconduct must be reported by instructors to CLAS according to these procedures. Graduate academic misconduct must be reported to the Graduate College according to Section F of the Graduate College Manual.

Artificial Intelligence (AI) Policies: Solutions to homework and examinations generated by AI tools are not allowed.

Student Collaboration on homework: The homework for this course is designed to help you master your knowledge related to the topics covered during lecture. As such, you may discuss on the homework problems with others or use online resources. However, please be aware that to master the skills needed for this class, practice is required and that to do well on the examinations you will need to work many of these problems multiple times without help. Be sure to test your knowledge by doing much of the homework on your own. Students are allowed to partially collaborate with others on homework through discussion for the most difficult problems. However, each student must turn in their own homework and it must not be a copy of someone else homework. Students are responsible for understanding this policy; if you have questions, ask for clarification. Word per word copies will not be tolerated. In extreme cases students may be requested to stop any kind of collaboration with other students.

Student Complaints: Students with a complaint about a grade or a related matter should first discuss the situation with the instructor and/or the course supervisor (if applicable), and finally with the DEO (Chair) of the department, school or program offering the course. Sometimes students will be referred to the department or program's Director of Undergraduate Studies (DUS) or Director of Graduate Studies (DGS). Undergraduate students should contact CLAS Undergraduate Programs for support when the matter is not resolved at the previous level. Graduate students should contact the CLAS Graduate Affairs Manager when additional support is needed.

Drop Deadline for this Course: You may drop an individual course before the drop deadline; after this deadline you will need collegiate approval. You can look up the drop deadline for this course here. When you drop a course, a "W" will appear on your transcript. The mark of "W" is a neutral mark that does not affect your GPA. To discuss how dropping (or staying in) a course might affect your academic goals, please contact your Academic Advisor. Directions for adding or dropping a course and other registration changes can be found on the Registrar's website. Undergraduate students can find policies on dropping CLAS courses here. Graduate students should adhere to the academic deadlines and policies set by the Graduate College.

UI Email: Students are responsible for all official correspondences sent to their UI email address (uiowa.edu) and must use this address for any communication with instructors or staff in the UI community. For the privacy and the protection of student records, UI faculty and staff can only correspond with UI email addresses.

Grading System and the Use of +/-: In assigning grades, the plus/minus grading system will be used. The A+ grade will be used only in extraordinary situations. Final grades will be awarded based on the following ranges:

Course Grades: The final grade will be based as follows:

1. There will be 2 tests during the semester, with each test to account for 22.5% of the course grade.
   • Midterm Exam 1: Wednesday March ??: 6:30-8:30PM in room ???.
   • Midterm Exam 2: Wednesday April ??: 6:30-8:30PM in room ???.
2. The final examination will account for 30% of the course grade.
3. Homework assignments will account for 20% of the course grade. The grade for your homework will be based on all the homeworks minus your worst 2 scores.

The 2 tests and final examination are open books and open notes examinations. Smartphones/computers are not allowed. Bring a simple scientific calculator, graphing calculators are fine.

Homework: Will be assigned approximately weekly. Presentation of your results is very important. Scratch paper will not be accepted. Do not expect good grades if your solution to a problem is poorly communicated. Like for everything, if you cannot explain something in great details, you certainly have not fully understood it. The importance of doing homework cannot be overemphasized, most of human people learn by doing, not only by watching and/or listening. Late homework may not be accepted, you need to request permission first or to provide a reasonable justification. Late homework is not accepted once a correction is given. Use of symbolic mathematical software to solve problems is not allowed.

Computer languages: The predominant programming languages used in numerical analysis are Matlab and Fortran. They are available on the Linux network in MLH (see below). Alternatives to Matlab are Octave and Scilab. For programming assignments, no other language will be accepted, except Python.

Linux computer accounts: Linux computer accounts are available on the Linux network in MLH (computer lab rooms B5). To access your Linux computer account remotely. you can use FastX, a graphical Linux virtual desktop environment remotely accessible in your web browser. As long as you have an active Hawk ID and you login at least once in the past year, your CLAS Linux account will remain active. If you fail to use your account in a year, you will receive three notices, and then your CLAS Linux account will be deleted. Also, once your Hawk ID becomes inactive, your CLAS Linux account will be deleted.

Date and Time of the Final Exam: The final examination date and time will be announced by the Registrar generally by the fifth week of classes and it will be announced on the course ICON site once it is known. Do not plan your end of the semester travel plans until the final exam schedule is made public. It is your responsibility to know the date, time, and place of the final exam. According to Registrar's final exam policy, students have a maximum of two weeks after the announced final exam schedule to request a change if an exam conflict exists or if a student has more than two exams in one day (see the policy here).

Grader: ???, office: ??? MLH, mailbox is in 15 MLH (MacLean Hall), e-mail: ???@uiowa.edu.

Course's College (Administrative Home)

• For undergraduate courses: The College of Liberal Arts and Sciences (CLAS) is the home of this course, and CLAS governs the add and drop deadlines, academic misconduct policies, and other undergraduate policies and procedures. Other UI colleges may have different policies.
• For graduate courses: The College of Liberal Arts and Sciences (CLAS) is the home of this course, and CLAS governs the policies and procedures for its courses. Graduate students, however, must adhere to the academic deadlines set by the Graduate College.

University Policies

• Free Speech and Expression: The University of Iowa supports and upholds the First Amendment protection of freedom of speech and the principles of academic and artistic freedom. We are committed to open inquiry, vigorous debate, and creative expression inside and outside of the classroom. Visit the  Free Speech at Iowa website for more information on the university's policies on free speech and academic freedom.
• Non-discrimination Statement: The University of Iowa prohibits discrimination in employment, educational programs, and activities on the basis of race, creed, color, religion, national origin, age, sex, pregnancy (including childbirth and related conditions), disability, genetic information, status as a U.S. veteran, service in the U.S. military, sexual orientation, gender identity, or associational preferences. The university also affirms its commitment to providing equal opportunities and equal access to university facilities. For additional information on nondiscrimination policies, contact the Senior Director, Office of Civil Rights Compliance, the University of Iowa, 202 Jessup Hall, Iowa City, IA 52242-1316, 319-335-0705,  ui-ocrc@uiowa.edu . Although not required, students have the option to share their pronouns and chosen/preferred names in class and through  MyUI. Instructors and advisors can find information about a student's chosen/preferred name in MyUI.
• Absences for Religious Holy Days: The university is prepared to make reasonable accommodations for students whose religious holy days coincide with their classroom assignments, test schedules, and classroom attendance expectations. Students must notify their instructors in writing of any such religious holy day conflicts or absences within the first few days of the semester or session, and no later than the third week of the semester. If the conflict or absence will occur within the first three weeks of the semester, the student should notify the instructor as soon as possible. See Policy Manual 8.2 Absences for Religious Holy Days for additional information.
• Accommodations for Students with Disabilities: The University is committed to providing an educational experience that is accessible to all students. If a student has a diagnosed disability or other disabling condition that may impact the student's ability to complete the course requirements as stated in the syllabus, the student may seek accommodations through Student Disability Services (SDS). SDS is responsible for making Letters of Accommodation (LOA) available to the student. The student must provide an LOA to the instructor as early in the semester as possible, but requests not made at least two weeks prior to the scheduled activity for which an accommodation is sought may not be accommodated. The LOA will specify what reasonable course accommodations the student is eligible for and those the instructor should provide. Additional information can be found on the SDS website.
• All University Course Policies and Resources for Students:: https://provost.uiowa.edu/student-course-policies