Optimization & Lagrange Multipliers

Understanding Constrained and Unconstrained Optimization

Prerequisites

Required Knowledge

  • Partial derivatives and their notation ($f_x, f_y, f_{xy}$ etc.)
  • Critical points in multivariable calculus
  • Second derivatives and the second derivative test
  • Basic system of equations solving
  • Understanding of contour plots and level curves

Key Terms Review

  • Critical Point: Where $f_x = 0$ and $f_y = 0$
  • Constraint: An equation $g(x,y) = 0$ that limits possible solutions
  • Level Curve: A curve where $f(x,y) = k$ for some constant $k$

Visual Understanding

Types of Critical Points

Local Minimum: Bowl Shape (z = x² + y²)

Local Maximum: Hill Shape (z = -(x² + y²))

Saddle Point (z = x² - y²)

Constrained Optimization Example

Level curves of f(x,y) = x² + y² with constraint x + y = 2

Part 1: Unconstrained Optimization

Finding Critical Points

For a function $f(x,y)$, find points where:

\[ f_x = 0 \quad \text{and} \quad f_y = 0 \]

The D-Test (Second Derivative Test)

If $f$ is $C^2$ near $(a,b)$, compute: \[ D = f_{xx}(a,b)\,f_{yy}(a,b) - \bigl(f_{xy}(a,b)\bigr)^2. \] Then:
  • If $D>0$ and $f_{xx}(a,b)<0$, \(f\) has a relative maximum\).
  • If $D>0$ and $f_{xx}(a,b)>0$, \(f\) has a relative minimum\).
  • If $D<0$, \(f\) has a saddle point\) at $(a,b)$.
  • If $D=0$, the test is inconclusive.

Examples: D-Test

Introductory Example: Simple Quadratic

Find and classify the critical point of $f(x,y)=x^2+y^2$

  1. Visualize: This is a bowl-shaped surface, like a paraboloid.

    Think of a ball rolling to the bottom of a bowl.

  2. Find partial derivatives:

    \[ f_x = 2x \quad \text{and} \quad f_y = 2y \]

  3. Set both equal to zero:

    \[ 2x = 0 \implies x = 0 \] \[ 2y = 0 \implies y = 0 \]

  4. Critical point is $(0,0)$. Now classify it:

    \[ f_{xx} = 2, \quad f_{yy} = 2, \quad f_{xy} = 0 \] \[ D = f_{xx}f_{yy} - (f_{xy})^2 = (2)(2) - (0)^2 = 4 > 0 \] \[ \text{Since } D > 0 \text{ and } f_{xx} > 0, \text{ this is a minimum} \]
Example 1: Classify the critical point of $f(x,y)=x^2+2xy-3y^2$.

  1. Compute partial derivatives:

    \[ f_x = 2x + 2y \quad \text{and} \quad f_y = 2x - 6y \]

  2. Set both partials to zero and solve the system:

    \begin{align*} 2x + 2y &= 0 \quad &\text{(Equation 1)}\\ 2x - 6y &= 0 \quad &\text{(Equation 2)} \end{align*} From Equation 1: $x = -y$ Substitute into Equation 2: \[ 2(-y) - 6y = 0 \implies -8y = 0 \implies y = 0 \] Therefore, $x = 0$ also. The critical point is $(0,0)$.

  3. Compute second partial derivatives:

    \[ f_{xx} = 2, \quad f_{yy} = -6, \quad f_{xy} = 2 \]

  4. Apply the D-Test at $(0,0)$:

    \begin{align*} D &= f_{xx}(0,0)\,f_{yy}(0,0) - \bigl(f_{xy}(0,0)\bigr)^2 \\ &= (2)(-6) - (2)^2 \\ &= -12 - 4 \\ &= -16 \end{align*}

    Since $D < 0$, the point $(0,0)$ is a saddle point.

Example 2: Find and classify extrema of $g(x,y)=x^3 -3xy +y^2$.
  1. $g_x=3x^2-3y,\;g_y=-3x+2y$. Solve $3x^2-3y=0,\;-3x+2y=0$.
  2. From $y=x^2$ and $y=\tfrac{3}{2}x$, get $x^2=\tfrac{3}{2}x\Rightarrow x( x-1.5)=0\). So $x=0$ or $1.5$.
  3. When $x=0$, $y=0$. When $x=1.5$, $y=2.25$.
  4. Compute second partials: $g_{xx}=6x,\;g_{yy}=2,\;g_{xy}=-3.$
  5. At $(0,0)$: $D=0\times2-(-3)^2=-9<0$ → saddle. At $(1.5,2.25)$: $g_{xx}=9, D=9\times2-9=9>0,g_{xx}>0$ → relative minimum.

Part 2: Optimization with Constraints

Lagrange Multipliers Method

For a function $f(x,y)$ subject to constraint $g(x,y)=0$:

\[ L(x,y,\lambda) = f(x,y) - \lambda g(x,y) \] \[ \text{Solve:} \quad L_x = 0, \quad L_y = 0, \quad L_\lambda = 0 \]

Geometric Intuition: Lagrange Multipliers

Step 1: Visualize Level Curves

Think of contour lines on a topographic map:

  • Each curve represents points where $f(x,y) = c$ for some constant $c$
  • The gradient $\nabla f$ points in the direction of steepest increase
  • The gradient is perpendicular to level curves

Step 2: Understand the Constraint

The constraint $g(x,y) = k$ is like a path we must follow:

  • We can only move along this path
  • $\nabla g$ is perpendicular to the constraint curve

Step 3: Find the Extremum

At an extremum point:

  • The level curve of $f$ is tangent to the constraint curve
  • $\nabla f = \lambda \nabla g$ (gradients are parallel)
  • $\lambda$ tells us how the gradients scale relative to each other

Examples: Lagrange Multipliers

Example 1: Quadratic Function with Linear Constraint

Find the extremum of $f(x,y)=x^2+4y^2$ subject to $2x+y=51$

  1. Form Lagrangian: \[ L(x,y,\lambda) = x^2+4y^2 - \lambda(2x+y-51) \]
  2. Find partial derivatives: \[ L_x = 2x - 2\lambda = 0 \] \[ L_y = 8y - \lambda = 0 \] \[ L_\lambda = -(2x+y-51) = 0 \]
  3. Solve system: \[ x = \lambda, \quad y = \frac{\lambda}{8} \] \[ 2(\lambda) + \frac{\lambda}{8} = 51 \] \[ \lambda = 24 \implies x = 24, y = 3 \]
  4. Evaluate: $f(24,3) = 612$ (minimum)

Example 2: Quadratic Function with Mixed Terms

Find the extremum of $f(x,y) = 3x^2 + 2y^2 - 2xy$ subject to $x + y = 14$

  1. Form Lagrangian: \[ L(x,y,\lambda) = 3x^2 + 2y^2 - 2xy - \lambda(x + y - 14) \]
  2. Find partial derivatives: \[ L_x = 6x - 2y - \lambda = 0 \] \[ L_y = 4y - 2x - \lambda = 0 \] \[ L_\lambda = -(x + y - 14) = 0 \]
  3. Solve system: \[ 6x - 2y = \lambda \] \[ 4y - 2x = \lambda \] \[ x + y = 14 \]
  4. Solution: $x = 6$, $y = 8$, $\lambda = 20$
  5. Evaluate: $f(6,8) = 140$ (minimum)

Example 3: Mixed Signs in Quadratic Terms

Find the extremum of $f(x,y) = 3y^2 - x^2$ subject to $4x + 2y = 88$

  1. Form Lagrangian: \[ L(x,y,\lambda) = 3y^2 - x^2 - \lambda(4x + 2y - 88) \]
  2. Find partial derivatives: \[ L_x = -2x - 4\lambda = 0 \] \[ L_y = 6y - 2\lambda = 0 \] \[ L_\lambda = -(4x + 2y - 88) = 0 \]
  3. Solve: \[ x = -2\lambda \] \[ y = \frac{\lambda}{3} \] \[ -8\lambda + \frac{2\lambda}{3} = -88 \] \[ \lambda = 12 \implies x = -24, y = 4 \]
  4. Evaluate: $f(-24,4) = -528$ (minimum)

Example 4: Three-Variable Optimization

Find the extremum of $f(x,y,z) = x^2 + y^2 + z^2$ subject to $x + 2y - 4z = 42$

  1. Form Lagrangian: \[ L(x,y,z,\lambda) = x^2 + y^2 + z^2 - \lambda(x + 2y - 4z - 42) \]
  2. Find partial derivatives: \[ L_x = 2x - \lambda = 0 \] \[ L_y = 2y - 2\lambda = 0 \] \[ L_z = 2z + 4\lambda = 0 \] \[ L_\lambda = -(x + 2y - 4z - 42) = 0 \]
  3. Solve system: \[ x = \frac{\lambda}{2}, y = \lambda, z = -2\lambda \] \[ \frac{\lambda}{2} + 2\lambda + 8\lambda = 42 \] \[ \frac{21}{2}\lambda = 42 \implies \lambda = 4 \]
  4. Solution: $x = 2$, $y = 4$, $z = -8$
  5. Evaluate: $f(2,4,-8) = 84$ (minimum)

Extreme Value Theorem

If $f$ is continuous on a closed, bounded region $D$, then $f$ attains an absolute maximum and minimum on $D$, either at interior critical points or on the boundary $\partial D$.

Examples: Extreme Value Theorem

Example 5: Find absolute extrema of $f(x,y)=x+y$ on the square $0\le x,y\le 1$.
  1. Interior: $f_x=1,f_y=1$ never zero, so no interior critical points.
  2. Boundary edges:
    • $x=0,\;0\le y\le1:\;f=y$, min $0$ at $(0,0)$, max $1$ at $(0,1)$.
    • $x=1,\;0\le y\le1:\;f=1+y$, min $1$ at $(1,0)$, max $2$ at $(1,1)$.
    • $y=0,\;0\le x\le1:\;f=x$, same as $x=0$ edge.
    • $y=1,\;0\le x\le1:\;f=x+1$, same as $x=1$ edge.
  3. Compare: absolute minimum $0$ at $(0,0)$; absolute maximum $2$ at $(1,1)$.

Interactive Visualizations

Common Mistakes to Avoid

1. Forgetting to Check Boundary Points

When using the Extreme Value Theorem, students often:

  • Only find interior critical points
  • Forget to check endpoints or corners
  • Miss absolute extrema on boundaries

2. Issues with Lagrange Multipliers

  • Writing incorrect constraint equations
  • Forgetting to set $L_\lambda = 0$
  • Not verifying solutions satisfy the constraint

3. Sign Errors in D-Test

  • Mixing up conditions for max/min
  • Forgetting to check $f_{xx}$ sign when $D > 0$
  • Incorrect computation of $f_{xy}$ terms

Real-World Applications

1. Economics & Business

Maximize profit subject to budget constraints:

\[ \text{Maximize } P(x,y) = px + qy \text{ subject to } ax + by = c \]

where:

  • $x,y$ are quantities of products
  • $p,q$ are unit prices
  • $ax + by = c$ is the budget constraint

2. Engineering Design

Minimize material cost for cylindrical container:

\[ \text{Minimize } A = 2\pi r^2 + 2\pi rh \text{ subject to } \pi r^2h = V \]

where:

  • $r$ is radius, $h$ is height
  • $A$ is surface area (cost)
  • $V$ is required volume

3. Physics

Find minimum potential energy in mechanical systems:

\[ \text{Minimize } U(x,y) \text{ subject to geometric constraints} \]

Summary

  • Find critical points via $f_x=f_y=0$.
  • Apply D-Test to classify local extrema.
  • Use Lagrange multipliers for constraints.
  • Check boundaries when using the Extreme Value Theorem.