# How to apply an R prediction model to very big data from SQL database in parallel.

I dont need to load the entire dataset into memory. In fact I only need 1 row at a time to apply a trained model, get the predicted response and put that response somewhere, possibly back into another table in the DB. Question is, how to do this efficiently. Of course the entire SQL based DB wont fit into RAM, but a few million rows of it will at a time. Say the entire DB is a billion rows. Is there any package which efficiently retrieves the data, applies a model or runs a function on the data and repeats in a parallel way, utilizing all cores, RAM (to minimize number of over the network pulls), etc? Thank you.

If your model is more complicated and you want to do the prediction using R functions then you might need to move your data into something like an ff object (see the ff package) which stores the data on disk and brings only parts into memory at a time and provides apply like functions to process the whole dataset in chunks.