P183 modbus RTU

[flashmark]

Initial version of a simple Modbus RTU plugin. This plugin handles a Modbus device as a set of holding registers. Up to 4 holding registers can be read into the 4 values of the plugin. The plugin number is 183 as agreed elsewhere.

[TD-er]

Can you look at the timing stats for this plugin you added?
I suspect the reading like this will take 100's of msec per register read.

Please take a look at what I did for the Eastron plugin, where I set a queue of registers to read.
This way the waiting time per call is nihil and thus not blocking the rest of the ESPeasy functionality.

My intention is to make all ESPEasy plugins using modbus use this approach to share access to the same Modbus bus.

Also the ESPEasySerial for (nearly) all plugins is using the same set of PCONFIG() for the serial config.
I did not check if you did this, so please make sure to use the same way as in the most recent plugins using a serial port.

[flashmark]

I have been busy for some time. Try to pickup the comments soon. Indeed the exchange takes some time and I will look into creating a queue. Eastron was my inspiration, but I did not look in most of the details as it was mainly device specific handling. Serial settings were copied from Eastron. To share the modbus I need some more insights how the sharing is intended. I have too little experience with some details in ESPEasy.

[flashmark]

A small introduction of myself. I started as embedded software developer using mainly C, but I am for quite some time software architect and not doing professional coding anymore. I am definitely not a C++ expert. I am working for a large company building complex machines. I like to pickup smaller embedded software projects and domotica. And I really like the way ESPEasy is set up.

I see some issues with the current P078 implementation. The modbus and Eastron device specific features are mixed over the various "layers". There is the "plugin" that takes care of the configuration and external data (variable, config, web representation). Then the "data_struct" to put the behavior of the plugin in a C++ friendly environment (away from .ino). And there is a "Eastron library" in SDM.cpp.

If understood you well the requirements are:

• Multiple plugin instances of different plugin types can share the same physical Modbus. (Use the same link)
• Multiple links should be possible in parallel
• Modbus access shall not hold the CPU during message exchange

The P078 implementation uses a queue that can manage multiple plugins in theory. For me it is unclear how it can differentiate between multiple links. The plugin defines the serial link, if there are multiple plugins connected it seems the last plugin that initializes defines the link properties. Is this desired behavior?

The queue knows it is handling SDM messages and delivers the received holding register directly into the uservar: UserVar.setFloat(it->taskIndex, it->taskVarIndex, value);
It is a fast way to handle the data, but it makes it impossible to retrieve other data from the Modbus device.

My proposal would be to split the code into the following classes:

• Plugin & optional data_struct to handle the user interaction as standard in ESPEasy
• A Modbus class that handles one link. This includes a queue and Modbus packet coding/decoding.
• A singleton Modbus "broker" that manages a list of Modbus links. The broker handles init/terminate requests from the plugins and tries to combine requests from multiple plugins in a single Modbus link.
• Existing ESPEasy serial link class responsible for the data transport over the selected serial link

Broker and link should be outside any plugin as they can be shared by multiple plugin classes. As they are both Modbus specific they can be in a single file sharing a header file.

I still need to think about the details how to exchange data and fit the queue neatly in the design. What has to be in and how does it return results. The Modbus link should be able to handle both the RTU binary and ASCII format. It should be able to handle other Modbus message types. Both option for future only :-)

One thing to consider:
Do we want to have the serial link specified through the plugin or add Modbus links as dedicated resources to the hardware page? My preference would be to keep it as is. Will make the broker a bit more complex, but we can manage that.

[TD-er]

Well hats off to you sir, as you seem to have a very good idea of the layers we have right now :)

Right now, I am working on another pull-request to do a complete rewrite for ESPEasy's WiFi/network implementation.
This does add a "Network" tab, just like the current "Devices" and "Controllers" tab we currently have.
And when I say "just like", it really is very similar.
So the first table is showing all network adapters and a short overview of its current state/config.

When clicking "Edit" on such an adapter, you get the specific settings, very similar to how controllers and tasks are being dealt with.

My next idea for a follow-up on this is to add a tab for "Interfaces" (or buses, name is not yet clear).
This way you can define interfaces like Modbus, SPI, I2C, 1-wire, etc. (maybe also extended GPIO pins)

Especially some of those interfaces which share a bus for multiple devices, need a handler to deal with all devices and pending transactions on the bus.
And also as you rightfully mentioned, some main (singleton) handler to handle all interfaces of the same type.
For example, when requesting a read or write on a modbus device, you must make sure you have completed this before something else is requested.

This idea is already implemented (in a bit too specific way) by keeping track of a list of registers to read and where to put the result.
This does work fine for Eastron devices, as it is all the same. You call for a register and interpret the result, then store the value somewhere.
However this already has some limitation as it only assumes float values. There are however some registers which do not return a float.

So to "fix" this, I imagine there might be some new PLUGIN_MODBUS_READ_RESULT call, which then should be implemented in those plugins which support Modbus communications.
Then in the event (ESPEasy EventStruct), there should be the taskindex set and probably some other values too and a pointer to the read data.
Those plugins then should request a read to the Modbus handler from the PLUGIN_READ call.
This way the read is already way more generic.

Then adding a main handler would probably be something for a next pull-request so we can think of a more generic way to manage modbus handlers.

The main advantage with this is that there is no longer a collision when accessing modbus devices on the same bus and there is no active blocking wait for a reply from the device, which may take quite some time.
For example the SenseAir S8 may take 200+ msec to reply and currently does actively wait.
The same for the ACU28x (or how those power meters are called...) and probably also for those PZEM meters.

[flashmark]

Looks good. By the way, I am not in any hurry to push my branch. Please continue the framework and let me know where I can contribute for something modbus specific. A suggesting is to remove the word MODBUS in the event and keep is a bit more abstract like BUS_READ_RESULT. This can then be any pending bus transaction. I think that I2C could also benefit.

If we add this callback trough an event and a central bus or link administration the singleton management layer will be very simple. The plugin know the bus type and index and the manager returns the associated bus object. Maybe do some admin to check how many active plugins are coupled to the bus; and do initialization termination when the first joins or the last leaves.

By the way, I saw a Modbus_RTU file. Is this where the new bus management stuff should grow?

[TD-er]

Not likely that this will remain the (file) structure.

For the WiFi/network rewrite, I'm introducing namespaces like ESPEasy/net/wifi

The idea of having a generic bus callback/event also seems OK, as long as the bus manager/handler does keep in mind which task may be expecting specific bus responses.
Like there are certain devices which can be used via various different bus types. For example the same sensor on I2C or SPI and/or UART.
But that's something to worry about later.

[flashmark]

Sorry it is still work in progress I wanted to set aside. I had some other duties and am just picking up this project. Will update it soon with a more crisp design. Keep in mind: I am not a C++ coder, any corrections and suggestions are welcomed.
Main design separation into several classes:

• Plugin: Unique for a device and implementing the interaction with ESPEasy.
• Modbus_device: Represents one modbus device and is coupled to a plugin. Does the modbus message (de)coding.
• Modbus_link: Represents a physical modbus link (serial link). One link can serve multiple modbus_devices.
• Modbus_mgr: Connects the devices and the links. This singleton object knows the devices and the links. Main task is to create a link when the first device wants to connect and connect a device to a link on request.
• Queue: Each link has a queue of pending modbus transactions.

Main issues to resolve:

• Serial port configuration is connected to a modbus_link. But I can only configure a plugin. Workaround: last device connecting to a link determines the properties. (If the ESPEasySerialPort type is different then it is a new link, still to think how to handle multiple "software serial")
• Transaction takes too much time to wait/poll. Instead we use a queue to prepare transactions (one message exchange). As a result we need some callback mechanism to return the results.
• For the device to link relation the is a real callback is implemented as a class function and the link knows the modbus_device that queued the transaction.
• For the plugin to device relation I don't know how to properly implement a mechanism that can feed back the uint16 registers from the modbus reply. Converting them to plugin output values blurs the responsibilities of the plugin and the modbus_device. This will make reuse much more difficult (like the P078 plugin).
• To keep the link busy it needs to poll the serial link for a complete reply. For now this is done by a ten_per_second trigger from each plugin. This is a bit heavy on CPU load. I am looking for a direct triggering of the link objects. The code should be prepared for that.

Is there a way to store design documentation with a plugin?

[TD-er]

Why not use some (weak) std::shared_ptr like structure for this?

[tonhuisman]

Are these already in mega?

My changes have been merged into mega, that's probably causing these merge conflicts... 🤔

[flashmark]

Simple fix if you know how to get to the conflicts :-)
There was a single merge conflict only...

[tonhuisman]

Here's a .diff file (zipped so GH will allow the upload). The extracted file should be placed in the root of your local repository, then from a terminal (best to open/use that from inside VSCode, shortcut: <ctrl>-<shift>-<backtick>) use the command:
git apply P183-Modbus_mgr-tonhuisman.diff

That should go without any issues.
Besides the updated configuration stuff, I made a couple of optimizations, mostly replacing the rather expensive strformat() calls with concat(), where you're only appending a number to a string.

P183-Modbus_mgr-tonhuisman.zip

[flashmark]

Here's a .diff file (zipped so GH will allow the upload). The extracted file should be placed in the root of your local repository, then from a terminal (best to open/use that from inside VSCode, shortcut: <ctrl>-<shift>-<backtick>) use the command: git apply P183-Modbus_mgr-tonhuisman.diff

That should go without any issues. Besides the updated configuration stuff, I made a couple of optimizations, mostly replacing the rather expensive strformat() calls with concat(), where you're only appending a number to a string.

P183-Modbus_mgr-tonhuisman.zip

It states "error: No valid patches in input (allow with "--allow-empty")". And when double clicking the diff it states "Already up to date". But I don't see modifications in mudbus_mgr.cpp showing how to use the new library.

I am missing a clue on how I can show multiple, separate, serial links on the same web page as I do for the Modbus interface(s). I would like to see your modified version.

[tonhuisman]

Hmm, not sure what went wrong, here's an updated diff:
P183-Modbus_mgr-tonhuisman-2.zip
I added a -2 to the .diff filename 😉

[flashmark]

Hmm, not sure what went wrong, here's an updated diff: P183-Modbus_mgr-tonhuisman-2.zip I added a -2 to the .diff filename 😉

Thanks a lot, working on it.

[flashmark]

I got it working with a minor duplicate variable definition fixed. Thanks again for the help.

I like the selection of the serial port types using a selection variable. For the user the "disabled" it is a bit unclear. It is shown, but cannot be selected. Would be nice it it was greyed out or even skipped. But I am not the HTML expert....

There is something strange in how the I2C settings are handled. RX & TX pins are drawn separately with addFormPinSelect() from the calling webform setup function like show_modbus_interfaces(). The I2C selection is drawn from within serialHelper_webformLoad(). This makes the interface between the Plugin_Helper_serial library and the using form setup code quite complex. Especially as there are several naming conventions involved for the pins and other values. Some documentation would be welcome to prevent undesired usage.

[flashmark]

I have a question about the I2C serial. Does it support automatic direction changing? I copied the DE/RE switching pin from other RS485 based plugins and the serial interface has some support for it. The UART seems to have some hardware support for this. But is this also supported for I2C and software serial?

For now the modbus setup page does support selection of the DE/RE pin for all serial port types, but it does not guarantee it works. I am using a serial to RS485 converter that does an automatic switching between TX and RX. I cannot test the pin-controlled switching.

I updated the pull request with the update to use the new serial library functions from tonhuisman. What is next to conclude this exercise. It grew a bit bigger that expected when I started to play with a simple Modbus device...

Also note that conversion of existing plugins will not be easy as the existing plugins include the setup of the serial link. That is now delegated to the Modbus link and the interfaces page.

What needs to be reviewed carefully:

• way the time base processing is hooked up to the scheduler. Now the scheduler knows there is a Modbus client based upon a #define. Form architecture point of view I would like to see a subscription mechanism. But the current pattern is also used for other task...
• Conditions to include the code. Should be well-aligned with ESPEasy standard patterns. I think it should be excluded (al least by default) on ESP8266 due to code size. The framework is only needed when at least one plugin requires it. For now only P183 can use it.

[tonhuisman]

The UART seems to have some hardware support for this. But is this also supported for I2C and software serial?

AFAICS, there's a TODO in the ESPEasySerial driver for the SC16IS752 to investigate if this can be enabled. The chip does seem to support that, so it should be possible to implement. OTOH, a similar TODO is in the code for SW-Serial, so there it also isn't implemented (yet).

You could, for now, remove the INCLUDE_SW_SERIAL and INCLUDE_I2C_SERIAL options from line 316 in Modbus_mgr.cpp, like the code I provided in the .diff file, as that wasn't an omission but on purpose... 🤔

• Conditions to include the code. Should be well-aligned with ESPEasy standard patterns.

Most parts are already in place from line 3160 in define_plugin_sets.h, and it looks like this define is already checked where needed

[tonhuisman]

The INCLUDE_SW_SERIAL and INCLUDE_I2C_SERIAL options don't support DE/RE or automatic RX/TX switching (yet), and I2C serial has a fixed buffer size of 64 bytes that can easily cause overflows on high-volume Modbus buses, so these ports should better not be used for Modbus, and thus not included here for now.

Also, to avoid weird behavior, they shouldn't be combined by adding (+) but or-ed (|), so a duplicate/alias value doesn't mess things up unexpectedly.

[TD-er]

Also SW-serial is really horrible on some ESP32's like the ESP32-C3.
SW Serial should be used as absolute last resort, so best to leave it out for this ModbusRTU feature.

[flashmark]

Thanks, the answers I was looking for. Will disable them.

[tonhuisman]

is in the logging ?

[flashmark]

Indeed. It is mend for debugging. I don't know how I can provide a table for debugging purposes in another way. Or do you only intend to attend to the spelling mistake.

By the way: I also don't know how I can best provide a holding register from the cache as result of a command.

[tonhuisman]

The convention for PLUGIN_GET_CONFIG_VALUE is to use a period (.) as the separator, not a comma (though some plugins support the comma as a fall-back).
const String cmd = parseString(string, 1, '.');

And using the standard unsigned int parser (that handles 0x hex and 0b binary notation):

uint32_t address{};
if (validUIntFromString(parseString(string, 2, '.'), address)) {
  uint16_t value = 0;
  ... the rest of your code up to and including success = true;
}

[flashmark]

Nice :-)

[flashmark]

I have some problems in the documentation that already existed for some time. The plugin overview shows [P183_status] instead of the defined status. Also the plugin sets are build with errors.

Building the documentation shows

PS C:\Users\mflesch\Documents\GitHub\ESPEasy\docs> .\make.bat html
Parsing substitutions for build sets...
Writing build sets overview to: source\Plugin\_plugin_sets_overview.repl
Traceback (most recent call last):
  File "C:\Users\mflesch\Documents\GitHub\ESPEasy\docs\builds_overview.py", line 208, in <module>
    generateBuildOverview('../Plugin/_plugin_sets_overview.repl')
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mflesch\Documents\GitHub\ESPEasy\docs\builds_overview.py", line 192, in generateBuildOverview
    output.write('      ":ref:`' + p + '_page`", "✓", "' + ('✓' if p in allBuilds_lb[b] else '') + '", "' + p + '"\n')
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mflesch\AppData\Local\Programs\Python\Python314\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u2713' in position 27: character maps to <undefined>
Running Sphinx v9.1.0
loading translations [en]... done
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
writing output... 
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 4 changed, 0 removed
reading sources... [100%] Reference/Command
source/Plugin/_plugin_sets_overview.repl:9: WARNING: The "csv-table" directive requires content; none supplied.

.. csv-table::
   :header: "Plugin name", "ESP32", "ESP8266", "Plugin number"
   :widths: 10, 3, 3, 3 [docutils]
C:\Users\mflesch\Documents\GitHub\ESPEasy\docs\source\Reference\Command.rst:722: CRITICAL: Problems with "include" directive path:
InputError: [Errno 2] No such file or directory: 'source/Plugin/P123_commands.repl'. [docutils]
looking for now-outdated files... none found

[tonhuisman]

The plugin overview shows [P183_status] instead of the defined status.

In _Plugin.rst you used square braces around the P183 status directives, once you fix that to have pipe symbols it'll look like this:

Not sure if DEVELOPMENT is the correct status, as we don't have a DEVELOPMENT build... probably better, and less prone to forgetting to update, by naming the actual builds it's supposed to go in. The docs will be published when the code is merged into mega, making it actually available, and the development moniker shouldn't be correct anymore 😉

[flashmark]

Thanks, Difficult to spot on my screen. I am not sure where to put this P183 plugin. It definitely does not fit on the ESP8266 builds. It is running fine for some time on my ESP32-C3. But I cannot guarantee is is production proof yet. What is the release strategy here?

[flashmark]

Still an issue with the documentation generation. Seems in my environment the "✓" is not recognized by codecs.charmap_encode(input,self.errors,encoding_table)[0]. This results in missing the overviews per collection. I did not touch the code so I expect this to be an issue in by build environment.

Other question is in which collection to put the P183 Modbus registers plugin. With all new Modbus facilities code behind it I would release it as "experimental", but we can assign it to a collection on ESP32 boards.

[TD-er]

You can of course put it in the MAX builds and since quite a lot of power meters and industrial sensors are using Modbus RTU, I guess Energy and Climate?

[TD-er]

About the checkmarks, you can compare it with the checkmarks used here: https://espeasy.readthedocs.io/en/latest/ESPEasy/ESPchips.html
Maybe also open the RST file in an editor like Notepad++ or some other editor which allows to also show the 'escape chars' to see if there is some odd byte in your text?

[flashmark]

About the checkmarks, you can compare it with the checkmarks used here: https://espeasy.readthedocs.io/en/latest/ESPEasy/ESPchips.html Maybe also open the RST file in an editor like Notepad++ or some other editor which allows to also show the 'escape chars' to see if there is some odd byte in your text?

Strange it is in a file not touched by me: \ESPEasy\docs\builds_overview.py
Function generateBuildOverview uses
output.write(' ":ref:' + p + '_page", "✓", "' + ('✓' if p in allBuilds_lb[b] else '') + '", "' + p + '"\n')
The checkmark shows as checkmark in Notepad++ and in VisualCode.
If I replace the "✓" with a normal character it works and I get a nice overview of the plugins by collection.

The mentioned Supported ESP Chips page only shows the red cross symbol, not the checkmark "✓". My generated code shows the same page with the red cross symbol.

I also copied the symbol in plain documentation code (.repl). Works fine.

I suspect something with language/character sets is wrong on my Windows machine or in the Python library. It is the Python interpreter that issues an error when calling function output.write().

For now I don't see it blocking for the Modbus delivery. Will put it in MAX for now.

[tonhuisman]

I suspect something with language/character sets is wrong on my Windows machine or in the Python library. It is the Python interpreter that issues an error when calling function output.write().

What Python version are you using? type python --version to find out, it should be something from 3.10 or newer.

You can also run the make html command from a pioarduino terminal session, that comes with Python 3.13+, by starting that terminal using the button in the VSCode bottom toolbar:

then cd docs and run the build: make html

Edit:
Hmm, this must be a Windows thing. I've been using WSL for so long now, for building ESPEasy, that this didn't show up until now 🤔
For some reason this character is missing from the cp1252.py table 🤓

Edit2:
Solution is to open the file for writing in utf-8 encoding,
builds_overview.py line 176: output = open(filepath, "w", encoding="utf-8")
I'll start a PR for this.

Oh, and btw, the checkmarks on the ESP Chips overview look just fine here 🤔