Skip to content

[Rules] Add string functions unescape, escape, parse and json #5600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tonhuisman wants to merge 4 commits into
base: mega
Choose a base branch
from
Open

[Rules] Add string functions unescape, escape, parse and json #5600

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5864b28
[Rules] Add string functions unescape, escape, parse and json
tonhuisman Aug 4, 2026
c39faca
[getJsonValue] Restore missing null return, update changelog, small c…
tonhuisman Aug 4, 2026
4e191fc
[Rules] Exclude new features for ESP8266 by default for build-size re…
tonhuisman Aug 5, 2026
dd07ba6
[Rules] Update Custom-sample.h
tonhuisman Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
190 changes: 190 additions & 0 deletions docs/source/Rules/Rules.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1100,6 +1100,196 @@ Usage: ``{lookup:<index>:<length>:<string_with_lookup_values>}``

``<string_with_lookup_values>``: The combined string with all lookup values padded to have the same length, f.e. ``"Off.Fan HeatCool"``, where all lookup values are of length 4. The index to retrieve the word ``Off.`` is 0, ``Fan`` (including a space) is index 1, etc. This string should be wrapped in quotes if it contains space or colon ``:`` character(s).

UnEscape and Escape
^^^^^^^^^^^^^^^^^^^

(Added: 2026/08/01, only available when String Variables feature is included in the build)

With ``UnEscape`` any ``\`` escape characters added before ``%``, ``[``, ``]``, ``{``, ``}``, ``(``, ``)`` and/or ``:`` is removed so the intended string value is made available, f.e. when sending to an external destination.

Usage: ``{unescape:<string-to-unescape>}``

As P037, and possibly other plugins, can generate events that include escaped JSON content, this helps to pass that on to other destinations.

With ``Escape`` in a string containing ``%``, ``[``, ``]``, ``{``, ``}``, ``(``, ``)`` and/or ``:``, any of these characters is prefixed with a ``\``, so it isn't interpreted as possible rules variables or functions.

Usage: ``{escape:<string-to-escape>}``

For passing received JSON or similar data as an argument to an event function.

Example:

.. code-block:: none

On testEscape Do
LetStr,j1,'\{"method"\:"setDo0","params"\:true\}'
LetStr,j2,`{unescape:[str#j1]}`
LogEntry,'j1=[str#j1], unescape=[str#j2], escape={escape:[str#j2]}'
Endon

Output:

.. code-block:: none

00:04:25.248 : (86376) Info | ACT : LetStr,j1,'\{"method"\:"setDo0","params"\:true\}'
00:04:25.257 : (86380) Info | ACT : LetStr,j2,`{"method":"setDo0","params":true}`
00:04:25.267 : (86216) Info | ACT : LogEntry,'j1=\{"method"\:"setDo0","params"\:true\}, unescape={"method":"setDo0","params":true}, escape=\{"method"\:"setDo0","params"\:true\}'
00:04:25.272 : > LogEntry,'j1=\{"method"\:"setDo0","params"\:true\}, unescape={"method":"setDo0","params":true}, escape=\{"method"\:"setDo0","params"\:true\}'
00:04:25.274 : (85772) Info | j1=\{"method"\:"setDo0","params"\:true\}, unescape={"method":"setDo0","params":true}, escape=\{"method"\:"setDo0","params"\:true\}
00:04:25.276 : > j1=\{"method"\:"setDo0","params"\:true\}, unescape={"method":"setDo0","params":true}, escape=\{"method"\:"setDo0","params"\:true\}
00:04:25.280 : > OK


Parse
^^^^^

(Added: 2026/08/02, only available when String Variables feature is included in the build)

Helps to 'parse out' parts of a string, based on a provided separator, based on the internal ``parseString()`` function. All input for the string processing functions is un-escaped before processing.

Usage: ``{parse:<param>:[<separator>]:<string-to-parse>}``

``param``: the nth option in the ``string-to-parse`` provided. n = 1-based

``separator``: choose the desired separator for parsing out the desired value. Defaults to ``,`` (comma) when left empty.

Example:

.. code-block:: none

On testParse Do
LetStr,p1,"!RFLink#AcuriteV2;ID=feb1;TEMP=80d5;HUM=65;BAT=OK;^^" // an input like received from RFlink
LetStr,RFDevice,{parse:2:#:{parse:1:;:[str#p1]}} // Split on ';', 1st element, then split on '#' > AcuriteV2
LetStr,RFID,{parse:2:=:{parse:2:;:[str#p1]}} // Split on ';', 2nd element, then split on '=' > feb1
LetStr,p2,{parse:2:=:{parse:3:;:'[str#p1]'}} // Split on ';', 3rd element, then split on '=' > 80d5
Let,temp,{and:0x[str#p2]:0x3ff}/10 // Calculate (nb. partially correct)
LetStr,p3,{parse:2:=:{parse:4:;:'[str#p1]'}} // Split on ';', 4th element, then split on '=' > 65
Let,hum,[str#p3]
LetStr,bat,{parse:2:=:{parse:5:;:'[str#p1]'}} // Split on ';', 5th element, then split on '=' > OK
LogEntry,'p2=[str#p2], p3=[str#p3]' // Check values
LogEntry,'[str#RFDevice]-[str#RFID],{"temp":[var#temp],"hum":%v_hum%,"bat":"[str#bat]"}'
Endon

Output:

.. code-block:: none

00:09:21.204 : (89356) Info | ACT : LetStr,p1,"!RFLink#AcuriteV2;ID=feb1;TEMP=80d5;HUM=65;BAT=OK;^^"
00:09:21.212 : (89392) Info | ACT : LetStr,RFDevice,{parse:2:#:!RFLink#AcuriteV2}
00:09:21.221 : (89436) Info | ACT : LetStr,RFID,{parse:2:=:ID=feb1}
00:09:21.229 : (89436) Info | ACT : LetStr,p2,{parse:2:=:TEMP=80d5}
00:09:21.236 : (89444) Info | ACT : Let,temp,213/10
00:09:21.244 : (89436) Info | ACT : LetStr,p3,{parse:2:=:HUM=65}
00:09:21.251 : (89524) Info | ACT : Let,hum,65
00:09:21.258 : (89436) Info | ACT : LetStr,bat,{parse:2:=:BAT=OK}
00:09:21.265 : (89436) Info | ACT : LogEntry,'p2=80d5, p3=65'
00:09:21.268 : > LogEntry,'p2=80d5, p3=65'
00:09:21.269 : (89324) Info | p2=80d5, p3=65
00:09:21.270 : > p2=80d5, p3=65
00:09:21.278 : (89308) Info | ACT : LogEntry,'AcuriteV2-feb1,{"temp":21.3,"hum":65,"bat":"OK"}'
00:09:21.283 : > LogEntry,'AcuriteV2-feb1,{"temp":21.3,"hum":65,"bat":"OK"}'
00:09:21.284 : (89092) Info | AcuriteV2-feb1,{"temp":21.3,"hum":65,"bat":"OK"}
00:09:21.286 : > AcuriteV2-feb1,{"temp":21.3,"hum":65,"bat":"OK"}
00:09:21.288 : > OK


Json
^^^^

(Added: 2026/08/02, only available when JSON Parse feature is included in the build)

Extract a value (numeric, text, bool) from a (valid) JSON object (string). All input for the string processing functions is un-escaped before processing.

Usage: ``{json:<attribute-to-retrieve>:[<asJson>]:<json-string>}``

``attribute-to-retrieve`` can use this format: ``level.attr[n].subattr``, multi-level supported, arrays[] are 0-based

``asJson``: Set to 1 (true) to return the value in JSON format (quoted strings, bool = true/false), default: 0 (false), some examples included below.

Returns 0/1 for a false/true ``bool`` value.

Returns a comma-separated list for array values without an index, see example. Returns ``unknown`` for unsupported value types, and a comma-separated name/value list for objects.

Example:

.. code-block:: none

On testJson Do
LetStr,j1,'\{"method"\:"setDo0","params"\:true,"values":\[1,2,3\],"args":\{"bla"\:"that","blip"\:false\},"array"\:\[\{"a":1,"b":2\},\{"a":10,"b":20\}\],"sub":\{"array":\[\{"c":1,"d":2\},\{"c":10,"d":20\}\]\}\}'
LetStr,j2,'{json:params::`[str#j1]`}'
LetStr,j3,'{json:values::[str#j1]}'
LetStr,j4,'{json:values[1]::[str#j1]}'
LogEntry,'json: {unescape:[str#j1]}'
LogEntry,'attributes: params=[str#j2], values=[str#j3], values[1]=[str#j4], bla={json:args.bla::[str#j1]},{json:args.bla:1:[str#j1]}, blip={json:args.blip::[str#j1]},{json:args.blip:1:[str#j1]}, array[1].b={json:array[1].b::[str#j1]}, sub.array[0].c={json:sub.array[0].c::[str#j1]}'
LogEntry,'array={json:array::[str#j1]}, sub={json:sub::[str#j1]}, (JSON)sub={json:sub:1:[str#j1]}'
Endon

The input, as assigned to ``j1``, is like it can be received from MQTT Import plugin. You can also use an incoming JSON payload, ``escape`` that (see above), and parse it. As JSON contains at least some ``{``, ``}`` and ``:`` characters, escaping is *required*, as these string functions use ``:`` as the parameter separator, and ``}`` as the function terminator.

Formatted JSON as used in the above example: (values marked with ``//`` are retrieved in the example)

.. code-block:: json

{
"method": "setDo0",
"params": true, //
"values": [ //
1,
2, //
3
],
"args": {
"bla": "that", //
"blip": false //
},
"array": [ //
{
"a": 1,
"b": 2
},
{
"a": 10, //
"b": 20
}
],
"sub": { //
"array": [
{
"c": 1, //
"d": 2
},
{
"c": 10,
"d": 20
}
]
}
}


Output:

.. code-block:: none

00:01:22.983 : (87572) Info | ACT : LetStr,j1,'\{"method"\:"setDo0","params"\:true,"values":\[1,2,3\],"args":\{"bla"\:"that","blip"\:false\},"array"\:\[\{"a":1,"b":2\},\{"a":10,"b":20\}\],"sub":\{"array":\[\{"c":1,"d":2\},\{"c":10,"d":20\}\]\}\}'
00:01:22.996 : (87712) Info | ACT : LetStr,j2,'1'
00:01:23.005 : (87572) Info | ACT : LetStr,j3,'1,2,3'
00:01:23.014 : (87600) Info | ACT : LetStr,j4,'2'
00:01:23.023 : (87116) Info | ACT : LogEntry,'json: {"method":"setDo0","params":true,"values":[1,2,3],"args":{"bla":"that","blip":false},"array":[{"a":1,"b":2},{"a":10,"b":20}],"sub":{"array":[{"c":1,"d":2},{"c":10,"d":20}]}}'
00:01:23.034 : > LogEntry,'json: {"method":"setDo0","params":true,"values":[1,2,3],"args":{"bla":"that","blip":false},"array":[{"a":1,"b":2},{"a":10,"b":20}],"sub":{"array":[{"c":1,"d":2},{"c":10,"d":20}]}}'
00:01:23.036 : (86528) Info | json: {"method":"setDo0","params":true,"values":[1,2,3],"args":{"bla":"that","blip":false},"array":[{"a":1,"b":2},{"a":10,"b":20}],"sub":{"array":[{"c":1,"d":2},{"c":10,"d":20}]}}
00:01:23.038 : > json: {"method":"setDo0","params":true,"values":[1,2,3],"args":{"bla":"that","blip":false},"array":[{"a":1,"b":2},{"a":10,"b":20}],"sub":{"array":[{"c":1,"d":2},{"c":10,"d":20}]}}
00:01:23.063 : (87132) Info | ACT : LogEntry,'attributes: params=1, values=1,2,3, values[1]=2, bla=that,"that", blip=0,false, array[1].b=20, sub.array[0].c=1'
00:01:23.067 : > LogEntry,'attributes: params=1, values=1,2,3, values[1]=2, bla=that,"that", blip=0,false, array[1].b=20, sub.array[0].c=1'
00:01:23.077 : (86724) Info | attributes: params=1, values=1,2,3, values[1]=2, bla=that,"that", blip=0,false, array[1].b=20, sub.array[0].c=1
00:01:23.096 : > attributes: params=1, values=1,2,3, values[1]=2, bla=that,"that", blip=0,false, array[1].b=20, sub.array[0].c=1
00:01:23.110 : (86748) Info | ACT : LogEntry,'array=a,1,b,2,a,10,b,20, sub=array,c,1,d,2,c,10,d,20, (JSON)sub={"array",[{"c",1},{"d",2},{"c",10},{"d",20}]}'
00:01:23.116 : > LogEntry,'array=a,1,b,2,a,10,b,20, sub=array,c,1,d,2,c,10,d,20, (JSON)sub={"array",[{"c",1},{"d",2},{"c",10},{"d",20}]}'
00:01:23.124 : (86356) Info | array=a,1,b,2,a,10,b,20, sub=array,c,1,d,2,c,10,d,20, (JSON)sub={"array",[{"c",1},{"d",2},{"c",10},{"d",20}]}
00:01:23.135 : > array=a,1,b,2,a,10,b,20, sub=array,c,1,d,2,c,10,d,20, (JSON)sub={"array",[{"c",1},{"d",2},{"c",10},{"d",20}]}
00:01:23.139 : > OK


IndexOf and IndexOf_ci
^^^^^^^^^^^^^^^^^^^^^^

Expand Down
2 changes: 2 additions & 0 deletions src/Custom-sample.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@
#define FEATURE_JSON_EVENT 0 // Generates an event with the values of a JSON repsonse of an HTTP call. Keys are stored in json.keys one key per line (e.g.: Body.Data.DAY_ENERGY.Values.1)
// #define FEATURE_SD 1 // Enable SD card support
// #define FEATURE_DOWNLOAD 1 // Enable downloading a file from an url
// #define FEATURE_JSON_PARSE 1 // Enable extended JSON parsing, disabled by default on ESP8266
// #define FEATURE_EXTENDED_STRING_FUNCTIONS 1 // Enable extra Rules string functions. Also required to enable FEATURE_JSON_PARSE on ESP8266, as it's disabled by default for ESP8266

#ifdef BUILD_GIT
# undef BUILD_GIT
Expand Down
40 changes: 25 additions & 15 deletions src/_P037_MQTTImport.ino
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
// This task reads data from the MQTT Import input stream and saves the value

/**
* 2026-08-02 tonhuisman: Add better support for JSON parsing, that handles multiple levels and arrays
* 2025-08-20 tonhuisman: Generate events with numeric values using the configured decimals setting.
* 2025-06-14 tonhuisman: Add support for Custom Value Type per task value
* 2025-01-12 tonhuisman: Add support for MQTT AutoDiscovery (not supported for MQTT Import)
Expand Down Expand Up @@ -58,10 +59,11 @@
# define P037_MAX_QUEUEDEPTH 150


bool MQTT_unsubscribe_037(struct EventStruct *event);
bool MQTTSubscribe_037(struct EventStruct *event);
bool MQTT_unsubscribe_037(struct EventStruct *event);
bool MQTTSubscribe_037(struct EventStruct *event);

# if P037_MAPPING_SUPPORT || P037_JSON_SUPPORT

String P037_getMQTTLastTopicPart(const String& topic) {
const int16_t lastSlash = topic.lastIndexOf('/');

Expand Down Expand Up @@ -470,7 +472,8 @@ boolean Plugin_037(uint8_t function, struct EventStruct *event, String& string)

// json filter check
if (checkJson && P037_data->hasFilters()) { // See if we pass the filters for all json attributes
do {
do
{
key = P037_data->iter->key().c_str();
Payload = P037_data->iter->value().as<String>();
# if P037_MAPPING_SUPPORT
Expand Down Expand Up @@ -521,7 +524,8 @@ boolean Plugin_037(uint8_t function, struct EventStruct *event, String& string)
if (checkJson && P037_data->hasFilters()) { // See if we pass the filters for all json attributes
P037_data->iter = P037_data->doc.begin();

do {
do
{
key = P037_data->iter->key().c_str();
Payload = P037_data->iter->value().as<String>();
# if P037_MAPPING_SUPPORT
Expand All @@ -541,7 +545,8 @@ boolean Plugin_037(uint8_t function, struct EventStruct *event, String& string)
# endif // P037_FILTER_PER_TOPIC
# endif // if P037_JSON_SUPPORT
{
do {
do
{
# if P037_JSON_SUPPORT

if (checkJson && (P037_data->iter != P037_data->doc.end())) {
Expand All @@ -552,13 +557,18 @@ boolean Plugin_037(uint8_t function, struct EventStruct *event, String& string)
if (!jsonAttribute.isEmpty()) {
key = jsonAttribute;

# if FEATURE_JSON_PARSE
Payload = getJsonValue(P037_data->root, key, false);
# else

if (key.indexOf('.') > -1) {
String part1 = parseStringKeepCase(key, 1, '.');
String part2 = parseStringKeepCase(key, 2, '.');
Payload = P037_data->doc[part1][part2].as<String>();
} else {
Payload = P037_data->doc[key].as<String>();
}
# endif // if FEATURE_JSON_PARSE
unparsedPayload = Payload;
int8_t jIndex = jsonIndex.toInt();

Expand Down Expand Up @@ -627,10 +637,10 @@ boolean Plugin_037(uint8_t function, struct EventStruct *event, String& string)
success = false;
break;
}
numericPayload = false; // No, it isn't numeric
doublePayload = NAN; // Invalid value
numericPayload = false; // No, it isn't numeric
doublePayload = NAN; // Invalid value
}
UserVar.setFloat(event->TaskIndex, x, doublePayload); // Save the new value
UserVar.setFloat(event->TaskIndex, x, doublePayload); // Save the new value

// Generate event for rules processing - proposed by TridentTD

Expand Down Expand Up @@ -665,9 +675,9 @@ boolean Plugin_037(uint8_t function, struct EventStruct *event, String& string)
addEscapeCharacters(tmp); // Add escape characters to avoid problems with rules processing if a JSON message is received

String RuleEvent = strformat(F("%s#%s=%s"),
getTaskDeviceName(event->TaskIndex).c_str(),
event->String1.c_str(),
wrapWithQuotesIfContainsParameterSeparatorChar(tmp).c_str());
getTaskDeviceName(event->TaskIndex).c_str(),
event->String1.c_str(),
wrapWithQuotesIfContainsParameterSeparatorChar(tmp).c_str());
P037_addEventToQueue(event, RuleEvent);
}

Expand All @@ -676,9 +686,9 @@ boolean Plugin_037(uint8_t function, struct EventStruct *event, String& string)

if (loglevelActiveFor(LOG_LEVEL_INFO)) {
addLog(LOG_LEVEL_INFO, strformat(F("IMPT : [%s#%s] : %s"),
getTaskDeviceName(event->TaskIndex).c_str(),
checkJson ? key.c_str() : getTaskValueName(event->TaskIndex, x).c_str(),
toString(doublePayload, ExtraTaskSettings.TaskDeviceValueDecimals[x]).c_str()));
getTaskDeviceName(event->TaskIndex).c_str(),
checkJson ? key.c_str() : getTaskValueName(event->TaskIndex, x).c_str(),
toString(doublePayload, ExtraTaskSettings.TaskDeviceValueDecimals[x]).c_str()));
}
# endif // if !defined(P037_LIMIT_BUILD_SIZE) || defined(P037_OVERRIDE)

Expand All @@ -695,7 +705,7 @@ boolean Plugin_037(uint8_t function, struct EventStruct *event, String& string)
} else {
String tmp = Payload;
addEscapeCharacters(tmp); // Add escape characters to avoid problems with rules processing if a JSON message is received

RuleEvent += wrapWithQuotesIfContainsParameterSeparatorChar(tmp);
}
P037_addEventToQueue(event, RuleEvent);
Expand Down
23 changes: 22 additions & 1 deletion src/src/CustomBuild/define_plugin_sets.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4394,6 +4394,27 @@ To create/register a plugin, you have to :
#endif
//-------------------End of HTTPResponseParser Section----------

#ifndef FEATURE_JSON_PARSE
#if defined(ESP32) && (defined(USES_P037) || FEATURE_JSON_EVENT) // and other JSON-parsing features
#define FEATURE_JSON_PARSE 1 // By default only on ESP32
#else
#define FEATURE_JSON_PARSE 0
#endif // if defined(USES_P037) || FEATURE_JSON_EVENT
#endif // ifndef FEATURE_JSON_PARSE

#ifndef FEATURE_EXTENDED_STRING_FUNCTIONS
#ifdef ESP32
#define FEATURE_EXTENDED_STRING_FUNCTIONS 1
#endif // ifdef ESP32
#ifdef ESP8266
#define FEATURE_EXTENDED_STRING_FUNCTIONS 0
#endif // ifdef ESP8266
#endif // ifndef FEATURE_EXTENDED_STRING_FUNCTIONS
#if defined(ESP8266) && FEATURE_JSON_PARSE && !FEATURE_EXTENDED_STRING_FUNCTIONS
#undef FEATURE_JSON_PARSE
#define FEATURE_JSON_PARSE 0
#endif // if FEATURE_JSON_PARSE && !FEATURE_EXTENDED_STRING_FUNCTIONS

#if !(defined(SOC_DAC_SUPPORTED) && SOC_DAC_SUPPORTED)
#ifdef USES_P152
#undef USES_P152
Expand All @@ -4414,7 +4435,7 @@ To create/register a plugin, you have to :
#if FEATURE_WIFI
#ifndef USES_NW001
#define USES_NW001
#endif
#endif
#ifndef USES_NW002
#define USES_NW002
#endif
Expand Down
Loading