9

Motivated by this top answer to the question: Why is convexity more important than quasi-convexity in optimization?, I am now hoping to understand why convex problems are easy to optimize (or at least easier than quasiconvex problems).

What are some of the most efficient algorithms for convex optimization, and why can't they be used effectively in quasiconvex problems?

Community
  • 1
Amelio Vazquez-Reina
  • 502
  • 3
  • 12
  • 1
    One extremely nice property is that if you draw a line/plane/hyperplane tangent to the graph of a convex function, the whole graph will lie to one side of the line, which doesn't work for quasiconvex functions. – Kirill Sep 24 '16 at 19:18

2 Answers2

7

You can try to apply a convex optimization algorithm to a non-convex optimization problem, and it might even converge to a local minimum, but having only local information about the function, you'll never be able to conclude that you've in fact found the global minimum. The most important theoretical property of convex optimization problems is that any local minimum (in fact, any stationary point) is also a global minimum.

Algorithms for global optimization of non-convex problems must have some sort of global information (e.g. Lipschitz continuity of the function) in order to prove that a solution is a global minimum.

To answer your specific question about why a convex optimization algorithm might fail on a quasi-convex problem, suppose that your convex optimization algorithm happens to be started at a "flat spot" on the graph of the objective function. There's no local information in the gradient to tell you where to go next. For a convex problem you could simply stop, knowing that you were already at a local (and thus global) minimum point.

Brian Borchers
  • 18,719
  • 1
  • 39
  • 70
  • I don't think this answers the question on convexity vs quasiconvexity. If the problem is just avoiding flat gradients one could assume that efficient convex methods work equally well for strictly quasiconvex functions, which I don't think is the case. – Amelio Vazquez-Reina Sep 24 '16 at 23:24
  • One interesting example is $y=x^{3}$. $x=0$ is a critical point, but the derivative is 0 at $x=0$. – Brian Borchers Sep 24 '16 at 23:34
  • There are of course lots of examples of points that are local minima of a quasi-convex function but not global minima. – Brian Borchers Sep 24 '16 at 23:35
  • Thanks. $x^3$ is a good example. That said, what if we restrict ourselves to quasi-convex functions with no critical points other than the global minimizer? Why is convexity specifically so valuable in optimization? Put another way, I understand that convex functions happen to have these nice properties, but wouldn't those properties extend to quasi-convex functions with no critical points other than the global minimizer? (do such functions have a name?). Wouldn't such functions be as easy to optimize as convex functions? – Amelio Vazquez-Reina Oct 04 '16 at 17:58
  • 1
    There's a standard theorem that says that if a descent method is used with a line search that satisfies the Armijo conditions then you get global convergence to a local minimum. (I've left out a few technical hypotheses here.) So yes, you could get global convergence to the minimum for your class of quasi-convex functions without non-optimal critical points. See for example Theorem 3.2 in the second edition of the text by Nocedal and Wright. – Brian Borchers Oct 04 '16 at 20:11
  • 1
    With respect to "as easy to optimize", you need to go beyond the issue of global convergence to a minimizer and consider rates of convergence. The analysis of many methods for convex optimization (e.g. quadratic convergence of Newton's method or the fast convergence of some recent accelerated first order methods for convex optimization) depends on convexity so these methods could fail on your class of quasiconvex functions. For example, a quasiconvex function might have a unique critical point but have points where the Hessian is singular, and this could break Newton's method. – Brian Borchers Oct 04 '16 at 20:18
  • 2
    Also keep in mind that its common for people speaking of convex optimization problems as being "easy to solve", they're generally talking about some classes of convex optimization problems (LP, Convex QP, SOCP, SDP, etc.) for which polynomial time algorithms exist and that can be solved easily in practice. More general convex optimization problems can be much harder to solve in practice. – Brian Borchers Oct 04 '16 at 20:25
  • To add to Brian Borchers point: if the problem is nonconvex, the Hessian is guaranteed to be singular somewhere. Convexity is equivalent to positive definiteness of the Hessian everywhere, so for a nonconvex function there must exist some region where the Hessian is not positive definite. The Hessian is singular along the boundary between the positive definite and not positive definite regions. To see this, consider tracking the eigenvalues of the Hessian as one traverses a path crossing between these two regions; as the eigenvalues go from negative to positive, they must pass through zero. – Nick Alger Oct 07 '16 at 20:05
5

Most of the best modern methods for large-scale optimization involve making a local quadratic approximation to the objective function, moving towards the critical point of that approximation, then repeating. This includes Newton's method, L-BFGS, and so on.

A function can only be locally well-approximated by a quadratic with a minimum if the Hessian at the current point is positive definite. If the Hessian is indefinite, then either

  1. The local quadratic approximation is a good local approximation to the objective function and is therefore a saddle surface. Then using this quadratic approximation would suggest moving towards a saddle point, which is likely to be in the wrong direction, or

  2. The local quadratic approximation is forced to have a minimum by construction, in which case it is likely to be a poor approximation to the original objective function.

(The same sort of issues arise if the Hessian is negative-definite, in which case it locally looks like an upside-down bowl)

So, these methods will work best if the Hessian is positive definite everywhere, which is equivalent to convexity for smooth functions.

Of course, all good modern methods have safeguards in place to ensure convergence when moving through regions where the Hessian is indefinite - E.g., line search, trust regions, stopping a linear solve when a direction of negative curvature is encountered, etc. However, in such indefinite regions the convergence is generally much slower, since full curvature information about the objective function cannot be used.

Nick Alger
  • 3,143
  • 15
  • 25
  • 1
    Disagree. Trust regions can handle indefinite quadratics. Even line search methods can by finding and doing line search on directions of negative curvature. On the other hand, if your algorithm is naked, with no trust region or suitable line search to protect you, then you'd be in trouble. But you can also be in trouble with such recklessness even with a strictly convex function. – Mark L. Stone Sep 24 '16 at 23:15
  • 3
    @MarkL.Stone Of course this is true and I debated mentioning it when writing the post. However, the point is that, yes you can make the method converge by doing special handling (as all good modern codes do), but the convergence is considerably slower. For example, a trust region method is equivalent to gradient descent if the trust region is small. – Nick Alger Sep 25 '16 at 01:22