RegEx Guardrail WARN and REDACT actions do not have any impact

[jw-proworks]

Which component is this issue related to?

Umbraco.AI (Core)

Which Umbraco AI version are you using? (Please write the exact version, example: 10.1.0)

1.0.0

Bug summary

The WARN and REDACT actions seem to have no effect on RegEx guardrails. Only the BLOCK action seems to cause the guardrails to be triggered.

Specifics

No response

Steps to reproduce

When set as follows:

the following prompt: "Is John Banta reputable?"

produces an error (which in the logs indicates the guardrail was invoked) from Umbraco Copilot dialog.

However, when changing the action to "WARN," no warning or error is received for the same prompt. In fact, a lengthy entry about why he is reputable is received.

Similarly, when using the "REDACT" action for the same prompt, nothing is redacted and a full response is given.

Expected result / actual result

I expected:

1. For the WARN action, a warning to be displayed somewhere in the response that indicated a guardrail had been triggered
2. For the REDACT action, all references to "John Banta" to be redacted since that was part of the regex match

Dependencies

No response

[jw-proworks]

The same is true for the WARN action on LLM Safety Judge guardrails - using this action produces no warning when using the same prompt that results in an error when the BLOCK action is used.

[mattbrailsford]

WARN only writes to the system logs so we don't show a particular warning in the UI, but I can look into the REDACT issue

[jw-proworks]

That makes sense - should the log indicate a warning or success? For me it is indicating success. I never see a warning in the log:

[mattbrailsford]

This fix is now released in Umbraco.AI 18.3.5 (v18) and 17.3.5 (v17), both available on NuGet.