% Generated by roxygen2 (4.1.1): do not edit by hand
% Please edit documentation in R/ddc.R
\name{csv2dframe}
\alias{csv2dframe}
\title{Load a CSV file into a distributed data frame.}
\usage{
csv2dframe(url, schema, delimiter = ",", commentCharacter = "#",
  hdfsConfigurationFile = paste(system.file(package = "hdfsconnector"),
  "/conf/hdfs.json", sep = ""), skipHeader = FALSE)
}
\arguments{
\item{url}{File URL. Examples: '/tmp/file.csv', 'hdfs:///file.csv'.

                     We also support globbing. Examples: '/tmp/*.csv', 'hdfs:///tmp/*.csv'.

                     When globbing all CSV files need to have the same schema and delimiter.}

\item{schema}{Specifies the column names and types.

               Syntax is: \code{<col0-name>:<col0_type>,<col1-name>:<col1_type>,...<colN-name>:<colN_type>}.

               Supported types are: \code{logical}, \code{integer}, \code{int64}, \code{numeric} and \code{character}.

               Example: schema='age:int64,name:character'.

               Note that due to R not having a proper int64 type we convert it to an R numeric. Type conversion work as follows:

               \tabular{ll}{
                   CSV type  |\tab R type    \cr
                   -         |\tab -         \cr
                   integer   |\tab integer   \cr
                   numeric   |\tab numeric   \cr
                   logical   |\tab logical   \cr
                   int64     |\tab numeric   \cr
                   character |\tab character
               }}

\item{delimiter}{Column separator. Example: delimiter='|'. By default delimiter is ','.}

\item{commentCharacter}{Discard lines starting with this character. Leading spaces are ignored.}

\item{hdfsConfigurationFile}{By default: \code{paste(system.file(package='hdfsconnector'),'/conf/hdfs.json',sep='')}.

                             Options are:
                             \itemize{
                                 \item webhdfsPort: webhdfs port, integer
                                 \item hdfsPort: hdfs namenode port, integer
                                 \item hdfsHost: hdfs namenode host, string
                                 \item hdfsUser: hdfs username, string
                             }

                             An example file is:

                                 \{ \cr
                                 "webhdfsPort": 50070, \cr
                                 "hdfsPort": 9000, \cr
                                 "hdfsHost": "172.17.0.3", \cr
                                 "hdfsUser": "jorgem" \cr
                                 \}}

\item{skipHeader}{Treat first line as the CSV header and discard it.}
}
\value{
A distributed data frame representing the CSV file.
}
\description{
Load a CSV file into a distributed data frame.
}
\section{Partitioning between executors}{


We generate as many partitions as files. If there are less files than executors we split each file further.

E.g. letâ€™s say we have 3 executors and we try to load /tmp/*.csv which expands to [/tmp/file1.csv (500MB) and /tmp/file2.csv (1MB)]. Initially we create 2 partitions (the number of files). As we have more executors (3) than partitions (2) we further divide the biggest file into 2. In the end we have 3 partitions (/tmp/file1.csv from 0 to 250MB, /tmp/file1.csv from 250MB to 500MB and /tmp/file2.csv.

If globbing is not used we only load one file which will be divided in as many chunks as executors.
}

\section{Details}{


There is a limitation in the case where the number of lines is less than the number executors.
In this case the load will fail. R's function \code{read.csv()} can be used instead.
}
\examples{
df <- csv2dframe(url=paste(system.file(package='HPdata'),'/tests/data/ex001.csv',sep=''), schema='a:int64,b:character')
}

