Exploring the clustering experiment of the AC BO Hackathon

[sgbaird]

There was surprising behavior from the corresponding notebook: https://github.com/AC-BO-Hackathon/project-26-multiple-context-bo/blob/main/cluster_experiment.ipynb

The key result of this study is that choosing clusters instead of providing the full available data definitely has an influence on the performance and might be worth a more in-depth investigation. Interestingly, it seems like the baseline actually performs worst in our experiments, although it has the maximum number of points available. This could have several reasons, like the chosen function not being a suitable one for this exercise, or the availability of relatively many points distracting the optimizer. Also, it might be the case that it might take more iterations for the baseline to "catch up" and overtaking the experiments using clusters.

In particular, notice how the "baseline" (which only contains an initial 30 datapoints, no additional clusters)

Compared with the transfer learning example from the BayBE docs [full example], the only difference is supposedly the use of clusters vs. the use of individual datapoints (and perhaps the amount of the data since clusters would have multiple data points (?)).

[mehradans92]

@sgbaird Rerunning the notebook, I can't produce the same result. Compare lines for 2, 5 and 10.

This is for 30 initial points.

[AVHopp]

I am getting a rather similar, but not identical plot with 0.9.0:

With 0.9.1 I get the following:

Not sure if there is undeterministic behavior involved?

[mehradans92]

Hi @AdrianSosic. Could you please let us know which version of baybe was used to generate this? Currently, I have 0.9.1 installed with pip.
I am seeing a much different behavior compared to what's in https://github.com/AC-BO-Hackathon/project-26-multiple-context-bo/blob/main/cluster_experiment.ipynb tho. See below:

Using 50 initial points:

Using 100 initial points:

[AVHopp]

I did these experiments during the hackathon, so I'll have a look into this. One thing for sure is that it was not done with version 0.9.1 since that version was not out at that time. Looking at the versions, it was probably 0.8.1 or 0.8.2, I will investigate.

[AVHopp]

Some updates from my side:

• I had trouble running the notebook with any version baybe<0.9.0. It would be great if you could elaborate on whether you can create plots with lower BayBE versions. From the errors I see, the issue might well be just related to my local environments or packages that are unavailable/different for a fixed version of BayBE, probably because the tests were done with an "intermediate" version.
• Note that you cannot directly compare the performance in this notebook with the one in the examples: The notebook uses a continuous search space and clusters within that, while the examples that we have on the website show a discretized search space.
• However, there is in fact weird behavior here, and we (mainly @AdrianSosic and me) are currently investigating this. I'm currently running some tests with a fresh 0.9.0 installation of BayBE and will share my results here. Reasons for different behavior when using baybe>=0.9.0 are probably some things that were changed in the backend in between.

Hope this makes some things a bit more clear. If not just ask, always happy to help 😄

[sgbaird]

Note that you cannot directly compare the performance in this notebook with the one in the examples: The notebook uses a continuous search space and clusters within that, while the examples that we have on the website show a discretized search space.
@AVHopp ah, thanks for catching! I had assumed the cluster experiment was discrete like the website, but hadn't dug in enough to the notebook code to see the distinction. I've mostly been focused on the website example.

For the motivation behind continuous vs. discrete benchmarking, it would be great to get some more context at #257 (comment).

[AVHopp]

I tried to give some explanation regarding how we use examples there. Do we want to take the discussion there and can we close this discussion then?

[sgbaird]

Thanks for clarifying! Yes, for this thread we can move to #257 (comment), but I think for the top-level discussion around the strange behavior with clustering experiments, it might make sense to keep this one open. Up to you though.

[AVHopp]

My question would be if we want to open a new, related discussion regarding "Behavior of BayBE for different versions" as this seems to be the point that is what causes the actual confusion here. If this is fine for everybody involved (in particular @mehradans92 ), I'd then just move the insights that we had from this discussion to a new one

[mehradans92]

Yeah. Sure :)

[AVHopp]

Here are the plots I got for `baybe 0.9.0":
30:

50:

100:

250:

[AVHopp]

A quick update on this:

• The "weird behavior" mentioned in one of the comments above is fixed by now - the issue was related to using TaskParameter in a purely continuous space, which is working by now.
• Also, we are currently implementing more benchmark functionalities: There is an open PR for bringing both Transfer Learning and non-TL benchmarks to BayBE. These can be run locally, but are also used internally for evaluation of BayBE. The (currently open) PR related to this is Add benchmarks #491 , and we expect this to be merged soon. So for benchmarking nowadays, we highly recommend to use the new benchmarks module and run this locally.