Phlex/FORM integration

[knoepfel]

This discussion serves as the central hub for discussing Phlex/FORM integration issues in preparation for Phlex v0.3.0.

[knoepfel]

@pcanal, @gemmeren, @wwuoneway, @aolivier23, and @knoepfel met on April 9, 2026 to discuss next steps in preparation for Phlex v0.3.0 (and beyond). My summary of the meeting is below.

A. For Phlex v0.3.0

1. Phlex v0.3.0 will support reading one FORM file as input (we will tackle reading multiple input files in FY27).
2. Phlex will enable the registration of providers that return type-erased data.
3. A goal is to retain the semantic distinction between framework drivers and data-product providers, while efficiently communicating data-cell indices (and other relevant metadata) between drivers and providers.
4. Right now, it is sufficient to use a data-cell index iterator which supplies one data-cell index at a time.
   • We do not need to support processing a range of data cells at one time.
5. The data-layer hierarchy will not be known until the file is opened (in contrast to the phlex::fixed_hierarchy behavior supported by Phlex v0.2.0):
   • Phlex will implement a driver interface that enables the opening of an input file
   • Upon opening that file, FORM will provide metadata that will be used by the framework to synthesize a provider node for each data product required from the input file.

B. After Phlex v0.3.0

1. The framework will need to be able to update the processed data-layer hierarchy when opening another input file.
2. The framework may support a running mode where all input files handled by a data-management system are declared to adhere to a specific hierarchy, thus avoiding unnecessary checking of the hierarchy contained in each file.
3. More than one input file may need to be open at the same time to support multi-threaded execution (transitions between input files).
4. More than one input file may need to be open at the same time to support (e.g.) data-MC overlay.

[pcanal]

I am confused in the ordering of 'v0.3.0 bullet 5' and 'after v0.3.0 bullet 2' as the later seem simpler to implement (Isn't the data-layer hierarchy determined by the graph described by the user configuration and the file either provide or do not provide the required input?)

[knoepfel]

Bullet B2 (sections now labeled) is meant to refer to a situation where a data-management system declares to Phlex what the hierarchy is for the files in that dataset. I can adjust the wording.

[knoepfel]

Wording for B2 has been adjusted.

[knoepfel]

@pcanal, @marcpaterno, and @knoepfel met on May 6 to discuss what Phlex needs to provide. Summary of the meeting:

Action items

• Define registration interface for providers that return type-erased data products
• Define driver interface that enables the opening of an input file
  • The driver emits one data-cell proxy (defined as a data-cell index with possibly additional metadata for efficiency purposes) at a time, exhibiting iterator-like behavior
  • Each provider returns a data product corresponding to the data-cell proxy it receives

Decisions

1. std::type_info will be used to request from FORM a data product with a specific type

   1. An "unrequited" data product will have a provider created for it
   2. For v0.3.0, an exception will be raised if more than one data product in the FORM file matches a data product given the specified product-query (including its std::type_info)

2. Phlex or FORM must be able to generate a reproducible name for a type if there is no product suffix associated with the data product.

Side notes:

• type_info (as part of product_base) is enough to express the type of the data product; using the demangled name is an alternative possibly and maybe be needed/required for the case of exchanging data products across process/platforms.

• Preservers and (likely) providers should be able to handle multiple data products produced by the same algorithm by simply handling the tuple. (Here there is 2 interface possibility either take the tuple directly or indirectly via template parameter packs or variadic functions).

Questions:

• For 0.3, does the provider returns just one type or can they also return a tuple? (Philippe guesses it is sufficient to return one type for 0.3).

[marcpaterno]

Here are my notes from the May 6 meeting.

• type_info is potentially a problem if the values need to cross process boundaries.
  My concern is that, because type_info hold implementation-specific information, the ABI of the libraries on both sides of the proess boundary would have to agree.
  This seems like it could cause trouble.
  I think we already know that remote protocols like xrootd will be important to DUNE.

• Creator nodes can return multiple data products from a single call, packed into a std::tuple.
  The current design of FORM has all elements stored in the tuple saved to a single container.
  To be able to read just one of the data products, and not all members of the tuple, we need to identify the correct "column" (I am not sure this is the correct use of the term column) that carries that tuple element.
  The information that will be provided by the "unrequited input" that we're looking to create a provider to satisfy will always include the concrete data product type desired.
  For the 0.3 release, it will also include a data layer with which the product is associated (recall that a specific product can be associated with multiple data layers).
  It may (but does not have to) include:

  1. a creator name
  2. a data-product concept
  3. a "suffix"

  We need the IO interface to allow Phlex to ask if there is a unique indexed family (https://framework-r-d.github.io/phlex-design/appendices/glossary.html#term-Indexed-family) matching the supplied constraints.
  If there is a unique one, we want to create a provider that will read them.
  If there is no such indexed family, we want to know that.
  If there are multiple, then the specification of the input is ambiguous, and we want to know that.
  We may want to know who the contenders are.

[pcanal]

< I think we already know that remote protocols like xrootd will be important to DUNE.

For xrootd, the exchange format is the ROOT file that is by design platform independent (and ROOT knows how to connect a type_info to the type names used inside the file)

[knoepfel]

Preamble

• The type phlex::product_ptr is a type alias for std::unique_ptr<phlex::product_base>.

• A provider function is a user-registered function that returns a data product, given a data-cell index. There are two kinds of provider-function signatures that are supported:

  // Function type for concrete data-product types (used by explicit providers)
  using int_provider_function_t = int(phlex::data_cell_index const&);
  
  // Function type for type-erased data-product types (used by implicit providers)
  using provider_function_t = phlex::product_ptr(phlex::data_cell_index const&);

  FORM will create provider functions that return a type-erased data product.

• We will have need of a class named phlex::provider_bundle, which the framework will use for registering providers from FORM:

  class provider_bundle {
  public:
    provider_bundle(product_family_specification, provider_function_t);
  
    product_family_specification specification() const;
    product_ptr get_product(data_cell_index const&) const;
  
  private:
    product_family_specification spec_;
    std::function<provider_function_t> provider_function_;
  };
  
  using provider_bundles = std::vector<provider_bundle>;

Phlex source base class

#include <memory>
#include <functional>

namespace phlex {	
  class source {
  public:
	// Each product_selector can correspond to one or more product families
	virtual provider_bundles create_providers(product_selector const&) = 0;
	virtual index_generator indices() = 0;
  };
}

Registering the FORM source

namespace {
  class form_source : public phlex::source {
  public:
    explicit form_source(phlex::configuration const& config);

    phlex::provider_bundles create_providers(phlex::product_selector const&) override;	
    phlex::index_generator indices() override;
	
  private:
    // ...
  };
}

PHLEX_REGISTER_SOURCE(g, config)
{
  return g.source<form_source>(config);
}

The form_source::indices() virtual function is, in fact, a coroutine that would look something like:

phlex::index_generator form_source::indices()
{
  auto job_index = data_cell_index::job();
  co_yield job_index;

  // Only go over the runs in the file
  for (unsigned int r : runs_in_my_file()) {
    auto run_id = job_id->make_child("run", r);
    co_yield run_id;
    ...
  }
}

We'll have to explore what that function should look like for more general cases.

Registering the FORM driver

The FORM driver is registered by creating a driver function (e.g., cells_to_process below) that receives the phlex::data_cell_yielder for the job, and the form_source.

void cells_to_process(phlex::data_cell_yielder const yield,
                      form_source const& form)
{
  for (phlex::data_cell_index_ptr const& index : form.indices()) {
	yield(index);
  }
}

PHLEX_REGISTER_DRIVER(d)
{
  return d.driver(cells_to_process);
}

Registering the FORM driver with configuration options

Configuration parameters can be passed to the driver function by using a lambda expression (e.g.):

void cells_to_process(phlex::data_cell_yielder const yield,
                      form_source const& form,
                      int const skip_every)
{
  for (phlex::data_cell_index_ptr const& index : form.indices()) {
      if (index.number() % skip_every == 0) {
        continue;
      }
      yield(index);
  }
}

PHLEX_REGISTER_DRIVER(d, config)
{
  // By default, skip every 10th data-cell index
  int const skip_every = config.get<int>("skip_every", 10);

  return d.driver(
    [skip_every](phlex::data_cell_yielder const yield, form_source const& form) {
      cells_to_process(yield, form, skip_every);
    });
}