Can qualification gating sit on BayBE's recommender API?

[TM289012]

Hi,

I'm a student who spent the summer building an open-source tool for ingredient replacement in formulated products: suggested experiment batches, staged qualification gates, Monte Carlo robustness, sign-offs tied to a hash of the evidence. https://github.com/TM289012/reformulation-assurance

The suggestion layer is currently GP plus random forest ranking over random candidates from the bounded mixture simplex. The docs label it exactly that, not BO. Long term, BayBE looks like the right brain for that layer.

Two questions:

1. Is wrapping your recommend loop in external gating and audit state an intended use, or would I be fighting the campaign abstraction?
2. Is the recommender API in v0.14.x stable enough for a student project to build against?

Happy to be told this is the wrong layer boundary & any other feedback.

[Scienfitz]

Hi @TM289012

Re 1

• I am not familiar enough with external gating and audit state, can you elaborate?
• Nonetheless I suspect the answer is yes. The campaign abstraction is actually optional in baybe, you can use it in a completely stateless mode by doing recommender.recommend(batch_size, searchspace, objective, measurements), the objects that go in can change rather flexibly between calls (the main thing is to ensure measurements are compatible with the searchspace). See the first bullet point here (the second one is about the stateful campaign)

Re 2
We are already moved beyond version 0.14 and there are substantial changes going on in searchspaces and recommenders, so honestly it's probably not that stable. But you can always pin a version if updating your code for a new baybe is not an option for you. Also, if you use baybe in a stateless mode there is usually very little baybe code you need to keep track of and maintain, we also deprecate most important objects (ie they will throw a warning but remain available).

Notes

• I agree its a good idea to use BO (includes uncertainty) instead of simple ML-based ranking (just predictive, falls in local minima)
• If you want to keep your two-step process of random candidate generation + subsequent ranking then you can use Searchspace.from_dataframe which you could use to create a searchspace identical to your self-generated candidates
• If you want to create a searchspace corresponding to the full space of possibilities you can use SubspaceDiscrete.from_simplex but there is currently still the danger that this crashes memory for too large spaces (see next point)
• We are actually currently upgrading baybe searchspaces to work lazily (don't crash memory), which will include a two-step process similar to your approach going via candidates (that are manifested in memory for ranking)
• 0.15 introduced a massive boost in optimization performance versus 0.14 so make sure to use that in case you use baybe
• In addition to the traditional way of modelling mixtures/formulations baybe offers a unique alternative called slot-based

[TM289012]

Thanks, this is exactly what I needed.

On your question about external gating and audit state: my app runs qualification stages (confirmation replicates, then process window, then supplier lots) and a candidate can't advance until real lab results pass that stage's criteria. The model only proposes experiments. Audit state means every prediction is frozen at recommendation time and scored against results later, and sign-offs are bound to a SHA-256 hash of the evidence they approved, so changes after signing are visible. Live demo: https://reformulation-assurance-demo.streamlit.app/ (public sandbox, one click to enter).

Based on your answers I pinned 0.15, kept my own candidate generation, and ran a first comparison: your recommender choosing from the same candidate pool my app builds, against my current GP plus RF ranking. Results are in a separate comment below, including the part where my first attempt handicapped BayBE unfairly. I also read the slot-based mixture examples; the emulsifier choice in my demo maps to exactly one slot, so that representation is on my roadmap.

[AVHopp]

Regarding how stable our code is: We take a lot of care in trying to add proper deprecation handling. That is, we usually try to ensure that there are at least several versions where "outdated" behavior still works and is usable, but raises a warning to the user. While this cannot be guaranteed for every feature, you can normally keep using older API versions for at least the next 2 to 3 releases which will then warn you about the feature being deprecated.

[TM289012]

Thanks, good to know. In today's benchmark script I got the new-style target constructors wrong, and the deprecated MATCH and MAX ones still worked, warnings and all, exactly as you described. Details in the results comment below.

[TM289012]

Here are the comparison results mentioned in my reply above. I'm posting the version with the fairness fixes built in, because my first pass had two design choices favoring my own arm, and I wanted them controlled, not just disclosed.

Setup (script in the repo: backtest_ranker_vs_baybe.py): semi-synthetic replay on my cosmetics demo project. 400 fresh candidates from my generator (legacy emulsifier locked to 0), identical for all arms. Ground truth is an oracle fit to the 81 completed real lots with residual-scale noise, run as two families (random forest and a neural net) so no arm wins by sharing a model family with the oracle. All arms start with the full 88-lot history. Success is the first pick in the pool's top 2% by true worst-spec margin. 5 seeds, censored at 40 rounds.

Arms: random; my app's ranking (GP plus RF, joint in-spec probability, pure exploitation); BayBE 0.15 stateless BotorchRecommender with a v1 scalarized single target; and BayBE with your native DesirabilityObjective, one normalized target per response.

oracle	arm	median rounds	min - max	censored
rf	random	30	7 - 38	1
rf	mine	2	1 - 2	0
rf	baybe (scalarized)	22	6 - 31	0
rf	baybe (desirability)	2	1 - 3	0
mlp	random	9	9 - 9	4
mlp	mine	4	1 - 5	0
mlp	baybe (scalarized)	13	4 - 17	0
mlp	baybe (desirability)	5	3 - 15	0

What I take from it: (1) My v1 scalarized target was the bottleneck, not your recommender. Native multi-target mode moved BayBE from 22 to 2 median rounds. My earlier caveat, confirmed by experiment. (2) At full strength your recommender and my ranking are basically tied on this task, under both oracle families. (3) Both beat random by a wide margin. (4) Regime caveat: all arms start from 88 informative measurements, friendly territory for pure exploitation. With sparse starting history I'd expect the BO machinery to pull ahead. That's the next experiment I want to run.

One API note: my guesses at the new-style target constructors were wrong, so the desirability arm ran on the deprecated MATCH and MAX constructors, which worked exactly as the deprecation policy described, warnings and all. If someone points me at the current 0.15 way to build a two-sided match target and a normalized one-sided target, I'll update the script.

Practical outcome for my app: pinning 0.15, and the integration seam (own candidate pool, SubspaceDiscrete.from_dataframe, stateless recommend) held up with no fighting. Thanks for the pointers.

[TM289012]

Oh yeah thanks for catching that, I did make a couple mistakes. match_triangular exists, I had just passed the bounds positionally instead of as cutoffs= (a ValueError). And the one-sided constructor is normalized_ramp, I had guessed normalize_ramp (an AttributeError, that method does not exist). Both work on 0.15.0:

NumericalTarget.match_triangular("viscosity_cp", cutoffs=(8000.0, 14000.0))
NumericalTarget.normalized_ramp("stability_score", cutoffs=(8.5, 10.0), descending=False)

I updated the benchmark script to use these, with the legacy forms kept as a fallback: TM289012/reformulation-assurance@da09920

Thanks for taking a look at it! Nothing else needed from my side.

[AdrianSosic]

OK, perfect 👌🏼. Just one recommendation from my side: Unless you have a specific reason why you need these special shapes, in most cases you are better off with match_bell and normalized_sigmoid, since they give you smooth gradients and are thus a simpler for the optimizer to handle.

[TM289012]

That sounds great. I'll switch to those next time I update the script.

[TM289012]

Following your suggestion, the benchmark now uses match_bell and normalized_sigmoid for the desirability targets, with the piecewise forms kept as a fallback: TM289012/reformulation-assurance@2a5b100

It made the BayBE arm stronger. On the RF oracle it now finds a top-2% candidate on the first round in all five seeds, up from a median of 2, and is slightly ahead of my ranker there. Mine stays slightly ahead on the MLP oracle. Updated results CSV is in the repo. Thanks for the feedback!

[AdrianSosic]

Glad it worked! Don't hesitate to reach out in case you have further questions 👌🏼