Sqoop tutorial 3 : Import arguments


argumentDescription
--appendAppend data to an existing dataset in HDFS
--as-avrodatafileImports data to Avro Data Files
--as-sequencefileImports data to SequenceFiles
--as-textfileImports data as plain text (default)
--boundary-query <statement>Boundary query to use for creating splits
--columns <col,col,col…>Columns to import from table
--directUse direct import fast path
--direct-split-size <n>Split the input stream every n bytes when importing in direct mode
--inline-lob-limit <n>Set the maximum size for an inline LOB
-m,--num-mappers <n>Use n map tasks to import in parallel
-e,--query <statement>Import the results of statement.
--split-by <column-name>Column of the table used to split work units
--table <table-name>Table to read
--target-dir <dir>HDFS destination dir
--warehouse-dir <dir>HDFS parent for table destination
--where <where clause>WHERE clause to use during import
-z,--compressEnable compression
--compression-codec <c>Use Hadoop codec (default gzip)
--null-string <null-string>The string to be written for a null value for string columns
--null-non-string <null-string>The string to be written for a null value for non-string columns

Comments

Post a Comment

Popular posts from this blog

Unix Tutorial 16 : Command Line Arguments

While running a command, the user can pass a variable number of parameters in the command line. Within the command script, the passed parameters are accessible using ‘positional parameters’.  These range from $0 to $9, where $0 refers to the name of the command itself, and $1 to $9 are the first through to the ninth parameter, depending on how many parameters were actually passed. Example: $ sh hello how to do you do Here $0 would be assigned sh $1 would be assigned hello $2 would be assigned how And so on … We will now look at some additional commands to process these parameters. #1) set Each process is also associated with a priority. This is used to ensure that the OS is able to fairly allocate time to various processing tasks. The ‘nice’ command can be used to reduced the priority of a process and thus be ‘nice’ to the other processes, i.e. $ nice <command> This line will run the specified command at a lower priority – by default, the priority wi...
Read more

Hive Tutorial 24 : MR vs Tez execution engines in Hive

Image
There are two execution engines available  to execute hive queries: 1. Mr 2. Apache Tez set  hive . execution . engine = mr set  hive . execution . engine = tez 1. Mr:          If we set execution engine as mr & executed the query, it will trigger map jobs first & store the result in local disks & the second job have to pick input from the local disk.Because of this the read/write operations show impact on its speed. 2. Apache Tez:         If we set execution engine as Tez & executed the query, it uses DAG's i.e.., instead of storing into local disk, it uses in-memory .Hence the read/write operations will be done more faster when compare to Mr. Tez internally triggers Vectorization which helps to increase its process speed. Observe the two images , the time taken for execution of the query using Tez engine is less than time taken by Mr engine.
Read more

Unix Tutorial 12 : Loop Types

Loops in Unix You may use different loops based on the situation. They are: #1) Unix For loop statement Example:  This program will add 1+2+3+4+5 and result will be 15 for i in 1 2 3 4 5 do sum=`expr $sum + $i` done echo $sum #2) Unix While loop statement Example:  This program will print the value of ‘a’ five times, from 1 to 5. a=1 while [ $a -le 5 ] do echo “value of a=” $a a=`expr $a + 1` done #3) Unix Until loop statement This program will print the value of ‘a’ two times from 1 to 2. a=1 until [ $a -ge 3 ] do echo “value of a=” $a a=`expr $a + 1` done While running these loops, there may be a need to break out of the loop in some condition before completing all the iterations or to restart the loop before completing the remaining statements. This can be achieved with the ‘break’ and ‘continue’ statements. The following program illustrates the ‘break’ operation: num=1 while [ $num -le 5 ] do read var if [ $var -lt 0 ] then ...
Read more