GlorotUniform Initializer unseeded

[scout792]

When running the "Inverse problem for the Lorenz system with exogenous input" example file, I get a warning about the GlorotUniform Initializer being unseeded.

/usr/local/lib/python3.12/dist-packages/tf_keras/src/initializers/initializers.py:121: UserWarning: The initializer GlorotUniform is unseeded and being called multiple times, which will return identical values each time (even if the initializer is unseeded). Please update your code to provide a seed to the initializer, or avoid using the same initializer instance more than once.

I plan to run an ablation study on a different inverse problem; however, I'd ensure I can run the same model with different seeds for that study.

I have added "dde.config.set_random_seed(101)" to the code immediately before the model.compile line. However, I get the same exact results for every run, regardless of the seed I use.

I am using the TensorFlow backend and am working in a google colab environment.

TensorFlow version: 2.20.0
TensorFlow Probability version: 0.25.0
Keras version: 2.20.0
DeepXDE version: 1.15.0

[vivek41-glitch]

I'd like to work on this issue. Could you please assign me?
The warning appears because GlorotUniform is called multiple times without a seed. The fix is to modify the initializer to accept a seed parameter and pass it to the Keras initializer.
I can implement this and update the relevant examples to show proper seeding.
Thanks!

[scout792]

I assumed it was an issue with how keras initializes the layers. I haven't dug into the source code; however, I assumed that the line of code that is needed is: "initializer = tf.keras.initializers.GlorotUniform(seed=user_provided_seed)." And that the user_provided_seed would be set with "dde.config.set_random_seed()".

[vivek41-glitch]

Thanks for confirming! Yes, that's exactly the approach I was thinking - pass a seed from dde.config.set_random_seed() into tf.keras.initializers.GlorotUniform(seed=...). I'll wait for the maintainer to assign me before submitting a PR

[vl-dud]

@scout792
If you don’t want to wait for the PR, you can apply a quick workaround like this:

import tensorflow.compat.v2 as tf
import deepxde.nn.initializers as dde_initializers

dde_initializers.INITIALIZER_DICT["Glorot uniform"] = tf.keras.initializers.GlorotUniform(seed=101)

[echen5503]

@vivek41-glitch I don't have triage access yet, so I can't assign you, but feel free to open a PR now and I'll review it when I see it.

[scout792]

@scout792 If you don’t want to wait for the PR, you can apply a quick workaround like this:

import tensorflow.compat.v2 as tf
import deepxde.nn.initializers as dde_initializers

dde_initializers.INITIALIZER_DICT["Glorot uniform"] = tf.keras.initializers.GlorotUniform(seed=101)

Thank @vl-dud ! That worked perfectly for what I needed.

[echen5503]

@scout792 I actually can't reproduce this issue, did you run the example exactly as-is?

[scout792]

@echen5503, I am running the example almost exactly as is. The only change was I had
dde.config.set_random_seed(101) immediately before the model.compile line.

I am using the TensorFlow backend and am working in a google colab environment.
TensorFlow version: 2.20.0
TensorFlow Probability version: 0.25.0
Keras version: 2.20.0
DeepXDE version: 1.15.0

[vivek41-glitch]

Hi maintainers, just checking in on this PR. The workaround works, but the proper fix (adding seed support to the initializer) is still valuable for long-term reproducibility.
All checks are passing and the code is ready for review. Please let me know if any changes are needed.

[echen5503]

@vivek41-glitch Yes, I am still paying attention to that PR; I was just waiting for the response of @scout792 to make sure I can reproduce the issue before suggesting the PR to be merged.

[echen5503]

@scout792 the following code should reconstruct the bug right?

"""Backend supported: tensorflow.compat.v1, tensorflow, paddle

Identification of the parameters of the modified Lorenz attractor (with exogenous input)
see https://github.com/lululxvi/deepxde/issues/79
"""
import re

import deepxde as dde
import matplotlib.pyplot as plt
import numpy as np
import scipy as sp
from scipy.integrate import odeint

# Generate data
# true values, see p. 15 in https://arxiv.org/abs/1907.04502
C1true = 10
C2true = 15
C3true = 8 / 3

# time points
maxtime = 3
time = np.linspace(0, maxtime, 200)
ex_input = 10 * np.sin(2 * np.pi * time)  # exogenous input

# interpolate time / lift vectors (for using exogenous variable without fixed time stamps)
def ex_func(t):
    spline = sp.interpolate.Rbf(
        time, ex_input, function="thin_plate", smooth=0, episilon=0
    )
    return spline(t)


# Modified Lorenz system (with exogenous input)
def LorezODE(x, t):
    x1, x2, x3 = x
    dxdt = [
        C1true * (x2 - x1),
        x1 * (C2true - x3) - x2,
        x1 * x2 - C3true * x3 + ex_func(t),
    ]
    return dxdt


# initial condition
x0 = [-8, 7, 27]

# solve ODE
x = odeint(LorezODE, x0, time)

# plot results
plt.plot(time, x, time, ex_input)
plt.xlabel("time")
plt.ylabel("x(t)")
plt.show()

time = time.reshape(-1, 1)

# Perform identification
# parameters to be identified
C1 = dde.Variable(1.0)
C2 = dde.Variable(1.0)
C3 = dde.Variable(1.0)

# interpolate time / lift vectors (for using exogenous variable without fixed time stamps)
def ex_func2(t):
    spline = sp.interpolate.Rbf(
        time, ex_input, function="thin_plate", smooth=0, episilon=0
    )
    return spline(t[:, 0:])


# define system ODEs
def Lorenz_system(x, y, ex):
    """Modified Lorenz system (with exogenous input).
    dy1/dx = 10 * (y2 - y1)
    dy2/dx = y1 * (28 - y3) - y2
    dy3/dx = y1 * y2 - 8/3 * y3 + u
    """
    y1, y2, y3 = y[:, 0:1], y[:, 1:2], y[:, 2:]
    dy1_x = dde.grad.jacobian(y, x, i=0)
    dy2_x = dde.grad.jacobian(y, x, i=1)
    dy3_x = dde.grad.jacobian(y, x, i=2)
    return [
        dy1_x - C1 * (y2 - y1),
        dy2_x - y1 * (C2 - y3) + y2,
        dy3_x - y1 * y2 + C3 * y3 - ex,
    ]


def boundary(_, on_initial):
    return on_initial


# define time domain
geom = dde.geometry.TimeDomain(0, maxtime)

# Initial conditions
ic1 = dde.icbc.IC(geom, lambda X: x0[0], boundary, component=0)
ic2 = dde.icbc.IC(geom, lambda X: x0[1], boundary, component=1)
ic3 = dde.icbc.IC(geom, lambda X: x0[2], boundary, component=2)

# Get the training data
observe_t, ob_y = time, x
observe_y0 = dde.icbc.PointSetBC(observe_t, ob_y[:, 0:1], component=0)
observe_y1 = dde.icbc.PointSetBC(observe_t, ob_y[:, 1:2], component=1)
observe_y2 = dde.icbc.PointSetBC(observe_t, ob_y[:, 2:3], component=2)

# define data object
data = dde.data.PDE(
    geom,
    Lorenz_system,
    [ic1, ic2, ic3, observe_y0, observe_y1, observe_y2],
    num_domain=400,
    num_boundary=2,
    anchors=observe_t,
    auxiliary_var_function=ex_func2,
)

plt.plot(observe_t, ob_y)
plt.xlabel("Time")
plt.legend(["x", "y", "z"])
plt.title("Training data")
plt.show()

# define FNN architecture and compile
net = dde.nn.FNN([1] + [40] * 3 + [3], "tanh", "Glorot uniform")
model = dde.Model(data, net)

dde.config.set_random_seed(101)
model.compile("adam", lr=0.001, external_trainable_variables=[C1, C2, C3])

# callbacks for storing results
fnamevar = "variables.dat"
variable = dde.callbacks.VariableValue([C1, C2, C3], period=100, filename=fnamevar)

model.train(iterations=60000, callbacks=[variable])

# Plots
# reopen saved data using callbacks in fnamevar
lines = open(fnamevar, "r").readlines()
# read output data in fnamevar (this line is a long story...)
Chat = np.array(
    [
        np.fromstring(
            min(re.findall(re.escape("[") + "(.*?)" + re.escape("]"), line), key=len),
            sep=",",
        )
        for line in lines
    ]
)

l, c = Chat.shape
plt.plot(range(l), Chat[:, 0], "r-")
plt.plot(range(l), Chat[:, 1], "k-")
plt.plot(range(l), Chat[:, 2], "g-")
plt.plot(range(l), np.ones(Chat[:, 0].shape) * C1true, "r--")
plt.plot(range(l), np.ones(Chat[:, 1].shape) * C2true, "k--")
plt.plot(range(l), np.ones(Chat[:, 2].shape) * C3true, "g--")
plt.legend(["C1hat", "C2hat", "C3hat", "True C1", "True C2", "True C3"], loc="right")
plt.xlabel("Epoch")

yhat = model.predict(observe_t)
plt.figure()
plt.plot(observe_t, ob_y, "-", observe_t, yhat, "--")
plt.xlabel("Time")
plt.legend(["x", "y", "z", "xh", "yh", "zh"])
plt.title("Training data")
plt.show()

[scout792]

@echen5503 The code you sent produces the bug, correct.
"/usr/local/lib/python3.12/dist-packages/tf_keras/src/initializers/initializers.py:121: UserWarning: The initializer GlorotUniform is unseeded and being called multiple times, which will return identical values each time (even if the initializer is unseeded). Please update your code to provide a seed to the initializer, or avoid using the same initializer instance more than once."

Additionally, I get a warning about tensorflow:AutoGraph with the code you sent. I have that warning by-passed in my code. However, the warning is:
"WARNING:tensorflow:AutoGraph could not transform <function at 0x7e33459182c0> and will run it as-is.
Cause: could not parse the source code of <function at 0x7e33459182c0>: no matching AST found among candidates:

coding=utf-8

lambda x, on: np.array([on_boundary(x[i], on[i]) for i in range(len(x))])"

[echen5503]

Ok, I was able to reproduce it after some trying. It seems like this warning does not appear for Keras 3 (tensor flow default), it will only show for Keras 2, which will happen if environment variable TF_USE_LEGACY_KERAS=1.

I'll review the PR from @vivek41-glitch now. Thanks!