CSVIngesterSocial.java reads entire sample file into memory

[zekizeki]

In
https://github.com/ONSdigital/rm-sample-service/blob/master/src/main/java/uk/gov/ons/ctp/response/sample/ingest/CsvIngesterSocial.java

A SampleUnit object is held in memory for each row in the CSV read. Larger sample files will blow the heap limits set and cause a crash.

[83naff]

From what I have read in the code, for each line in the file it is getting the attributes then returning them within each iteration of the for loop. Although the local variables in parseLine are destroyed, we are copying them, which would increase memory. What we can do at least is to reduce the copious amount of copying incurred.

[mfcrocker]

Hi Rob!

Though this is definitely something that will happen over a certain size, we've tested loading the biggest sample we've got (111k for ASHE) as part of the performance team's work and it loaded fine.

This is something that will want a refactor one day, but as it stands there's no business need to go >110k that I'm aware of.