[Bug]: <Title> Vectorizer Chunking

[samimiashkan]

What happened?

Hello.. I noticed, when I run the vectorizer (it is a docker container), it reads 50 records at the time and creates chunks.. but when it gets a field with longer content, it increases the no. of chunks (which is normal.. It should do it like that) but it keeps the higher rate of chunking (50 -> 100, 50 -> 110 , 50 ->130 ....) and continues. It does not come down anymore. when I stop the container and start it again, it goes back to lower rate chunks (50 -> 56, 50 ->61 ...) until the next longer field. is this a bug? Please review.

pgai extension affected

0.9.2 Image version.

pgai library affected

No response

PostgreSQL version used

17

What operating system did you use?

Debian

What installation method did you use?

Docker

What platform did you run on?

On prem/Self-hosted

Relevant log output and stack trace

How can we reproduce the bug?

I'm creating embeddings for over 2,3 mil records. the chunk rate of 50 -> ~60 seems to be normal.. but I think it can easily be reproduced with any type of data. Jut make sure between your record,  you have longer text for creating the embedding..

For embedding,  I'm using openAI "text-embedding-3-small", 1536.

Are you going to work on the bugfix?

None