You will have a local machine with 512GB of memory, so feel free to load the data in memory if it makes sense or if it fits (!) You will also have terabytes of memory in our Spark cluster that is not shared by anyone