3

I'm calculating the gradient of a function with a symbolic math library called theano. Then I'm using gradient descent to find the minimum of the function.

I'd like to prove that the minimum is a global minimum. How can I prove that my function is convex?

siamii
  • 141
  • 3

3 Answers3

8

There are many ways of proving that a function is convex:

  • By definition
  • Construct it from known convex functions using composition rules that preserve convexity
  • Show that the Hessian is positive semi-definite (everywhere that you care about)
  • Show that values of the function always lie above the tangent planes of the function
Geoff Oxberry
  • 30,394
  • 9
  • 64
  • 127
4

Unless you know something about the properties of the function (e.g., whether it's a quadratic polynomial, monotonic, etc), you can not experimentally determine whether a function is convex. You need to limit your question to a smaller subset of functions.

Wolfgang Bangerth
  • 55,373
  • 59
  • 119
0

You asked how to prove a function is convex---but it looks like your real question is how to prove that your local minimum is global. There are plenty of non-convex functions whose local minima are also global, and sometimes, you can even prove it :-)

One option you might consider is to employ a branch-and-bound global optimization approach. That's a systematic way for exploring the entire region of interest. In theory this could be very expensive, but in practice, you can often prune away large fractions of the region of interest, saving you a lot of computation.

Here are some slides and lecture notes by Stephen Boyd and Jacob Mattingley on the topic. To use this, you not only need to be able to compute local minima of your function (something you currently have) but the ability to compute good lower bounds for the function over arbitrary regions.

If you think this approach may have merit for your application, give us a little more detail about your problem (by editing your question) and perhaps I can fill in some more detail.

Michael Grant
  • 2,062
  • 11
  • 24
  • Branch-and-bound for nonconvex optimization with continuous decision variables requires at least an interval arithmetic library. The OP is using Python; I don't think there are many Python interval arithmetic libraries (mpmath does not have enough interval arithmetic features). They may need to switch to a different language to do nonconvex optimization. – Geoff Oxberry Apr 21 '13 at 01:41
  • This is not the case if you have a lower bound function. – Michael Grant Apr 21 '13 at 02:48
  • To be fair, what I am proposing here involves maintaining intervals explicitly instead of using interval arithmetic to handle them implicitly. So in a sense that's interval computation, just of a different sort. I am assuming that the function computations themselves can be treated as accurate. – Michael Grant Apr 21 '13 at 02:54
  • 1
    In order for branch-and-bound to converge, if memory serves, your family of lower bound functions needs to have certain properties upon partition refinement (I believe that as the partition diameter goes to zero, the lower bound functions should approach the functions being bounded), and you need to be able to generate these relaxations over arbitrary intervals. The burden would be on the OP to prove their relaxations yielded a convergent branch-and-bound procedure, in which case, a good reference would be the book by Horst and Tuy, "Global Optimization: Deterministic Approaches". – Geoff Oxberry Apr 21 '13 at 08:32
  • The reason I would recommend an interval arithmetic-based approach is because there are families of relaxations for which convergent branch-and-bound procedures have been proven, such as the McCormick relaxations used by Barton (Disclaimer: I worked for Barton) or $\alpha$BB relaxations developed by Floudas. – Geoff Oxberry Apr 21 '13 at 08:35
  • Yes, good points about the properties of your lower bound function. I figured that details such as these could wait to see if the OP expanded on the description of his function. But you do have more experience in these matters than I! Thank you for the input. – Michael Grant Apr 21 '13 at 12:20