Unix Tutorial 16 : Command Line Arguments

While running a command, the user can pass a variable number of parameters in the command line.
Within the command script, the passed parameters are accessible using ‘positional parameters’.  These range from $0 to $9, where $0 refers to the name of the command itself, and $1 to $9 are the first through to the ninth parameter, depending on how many parameters were actually passed.
Example:
$ sh hello how to do you do
Here $0 would be assigned sh
$1 would be assigned hello
$2 would be assigned how
And so on …
We will now look at some additional commands to process these parameters.
#1) set
Each process is also associated with a priority. This is used to ensure that the OS is able to fairly allocate time to various processing tasks. The ‘nice’ command can be used to reduced the priority of a process and thus be ‘nice’ to the other processes, i.e.
$ nice <command>
This line will run the specified command at a lower priority – by default, the priority will be reduced by 10. The command also takes a parameter that can be used to use a different level of ‘niceness’.
Example:
$ nice -20 ls
This command runs ‘ls’ with the priority reduced by 20.
It is also possible to increase the priority with a negative ‘niceness’. However, this needs superuser permission.
When a terminal or login session is closed, it sends the SIGHUP signal to thr child processes. By default, this signal will cause the child processes to terminate. The ‘nohup’ command can be used to allow commands to continue running even when the login session is terminated.
Example:
$ nohup soft file1 > file2
With this command, sorting of file1 and saving in file2 process will continue even if we have logged out of the system.
The ‘kill’ command can be used to terminate any of the processes depending on permissions.
Example:
$ kill [options] <pid>
This command will terminate a process with the process id <pid>. The PID of a process can be obtained using the ‘ps’ command.
This ‘at’ command is used to execute commands at a particular date and time in the future.
Example:

$ at 8pm



sort file1>file2

Unix Debugging

Unix provides a number of mechanisms to help find bugs in your command scripts. These mechanisms can be used to view a trace of what is being executed i.e. the sequence in which commands are executed. The trace can be used to understand and verify the logic and control flow of the script.
=> set -v
verbose mode: Setting this option before running a command will ensure that the command that will be executed is printed to stdout before it is actually executed.
=> set -x
execution trace mode: Setting this option will display each command as it is executed along with its arguments.
=> set -n
no-exec mode: Setting this option shows any errors without actually running any commands.

Comments

Post a Comment

Popular posts from this blog

Hive Tutorial 31 : Analytic Functions

Image
Hadoop Hive analytic functions compute an aggregate value that is based on a group of rows. A Hadoop Hive HQL analytic function works on the group of rows and ignores the NULL in the data if you specify Hadoop Hive COUNT Analytic Function Returns number of rows in query or group of rows. Syntax: COUNT(column reference | value expression | *) over(window_spec) For Example; select pat_id, dept_id, count(*) over (partition by dept_id order by dept_id asc) as pat_cnt from patient; at_id dept_id pat_cnt 6 111 4 2 111 4 5 111 4 1 111 4 4 222 3 5 222 3 3 222 3 7 333 1 8 444 1 Hadoop Hive SUM Analytic Function Just like count function, sum Hive analytic function is used to compute the sum of columns or expression. Sum analytic function is used to compute the sum of all rows of table or rows within the groups. Syntax: SUM(column | expression) OVER( window_spec) For example: Calculate sum insured amount of all patients within each department...
Read more

Hive Tutorial 37 : Performance Tuning

Performance plays key role in big data related projects as they deals which huge amount of data. So when you are using Hive if you keep few things in mind then we can see dramatic change in the performance.  Performance tuning in hive: Partitions Bucketing File formats Compression Sampling Tez Vectorization Parallel execution CBO Partitions : The concept of partitioning in Hive is very similar to what we have in RDBMS. A table can be partitioned by one or more keys. This will determine how the data will be stored in the table. For example, if a table has two columns, id, name and age; and is partitioned by age, all the rows having same age will be stored together. So when we try to query based on age range, then hive will retrieve the data by going into particular folders instead of parsing through whole data. /hdfs/user/tablename/age/10 /hdfs/user/tablename/age/11 Bucketing : Bucketing is more efficient for sampling,data will be segre...
Read more

How to change sqoop saved job parameters

At the initial stage I thought it was impossible to reset the incremental.last.value in a sqoop job. After creating a sqoop job for incremental import of data from sqlserver to hadoop, there comes a request to re-dump the data. Although, flushing the previously ingested data in hadoop seems to be effortless using the command hdfs dfs -rm /location/to/dir/part*  the tough one comes after trying to execute the sqoop job again. The job executes without pulling any record because it sees no increment in the number of records. So, instead of removing the sqoop job and creating another one afresh, the way to go is to reset the incremental.last.value to 0.  And this can be done by changing the value of the last record in the sqoop metastore. The steps involves: Navigate to the home directory using cd ~ Locate the sqoop metastore using ls -a You'll see a dir named .sqoop, cd into it with cd .sqoop vi metastore.db.script or nano metastore.db.script Scroll all the way down to s...
Read more