In the "Black-Box Variational Inference" section of the VAE notes:
We first do per-sample optimization of q by iteratively applying the update
λ(i) ← λ(i) + ∇̃ λ ELBO(x(i); θ, λ(i))
We then perform a single update step based on the mini-batch
θ ← θ + ∇̃ θ ∑i ELBO(x(i); θ, λ(i))
If I understood correctly, x(i) is the ith sample from a batch B of the dataset D, and λ is a vector of parameters of the distribution "qλ(z)". What is λ(i)?
Is it the ith parameter of λ? That would imply that the length of B is equal to the dimension of the λ--if so, it's unclear to me why they would be equal.
Another possibility is that λ(i) is the ith update to λ. If so, perhaps it would be better rewritten like this:
λ(i+1) ← λ(i) + ∇̃ λ ELBO(x(i); θ, λ(i))
But if that's the case, then it's unclear to me why it appears in the θ update:
θ ← θ + ∇̃ θ ∑i ELBO(x(i); θ, λ(i))
Apologies if I've missed something obvious here. Also, thanks for notes--they've been very helpful!
In the "Black-Box Variational Inference" section of the VAE notes:
If I understood correctly, x(i) is the ith sample from a batch B of the dataset D, and λ is a vector of parameters of the distribution "qλ(z)". What is λ(i)?
Is it the ith parameter of λ? That would imply that the length of B is equal to the dimension of the λ--if so, it's unclear to me why they would be equal.
Another possibility is that λ(i) is the ith update to λ. If so, perhaps it would be better rewritten like this:
But if that's the case, then it's unclear to me why it appears in the θ update:
Apologies if I've missed something obvious here. Also, thanks for notes--they've been very helpful!