Hive Tutorial 19 : Hive UDF

Beyond the multitude of string functions, the usual mathematical and date functions are available as well as conditionals and collections in form of arrays and maps. The most powerful functionality though is the ability for users to write their own functions to extend Hive.
Traditionally, many map-reduce programs had little original logic. They emerged from the need to execute a small piece of logic on a huge amount of data. The resulting Java programs were a lot of scaffolding to access and move data around. Today, in many cases Hive can do away with trivial map-reduce programs for ETL and analytics in favour of simple queries as we have seen.
There are a few cases where additional functionality beyond what Hive offers is needed. Reading and writing data in an exclusive format is one case. Hive can be extended to support any format by writing a SerDe, which is short for Serializer/Deserializer.
Other cases are; transformation of one row value into another one, which can be added with UDFs (User Defined Function); transformation of multiple row values into one, which can be added with UDAFs (User Defined Aggregate Functions); transformation of one row value into many, which can be added with UDTFs (User Defined Table Functions).
The method to add these is to use the same programming model Hive uses to implement the functions it ships with. These functions are in fact themselves just UDF/UDAF/UDTFs and the only difference to user functions are that they are shipped with Hive. A user can add them by writing the functions in Java, compiling them into a JAR and loading the JAR in Hive. From that point on the user’s functions are accessible in Hive queries just like any other function, e.g. LENGTH (UDF), COUNT (UDAF), or EXPLODE (UDTF).
Let’s explore the LENGTH example, an UDF, which is available in Hive. The query SELECT LENGTH(col1) FROM table; applies the function LENGTH on strings like ‘abcdef’ and returns ‘6’, for example. 

package org.apache.hadoop.hive.ql.udf;

import org.apache.hadoop.hive.ql.exec.Description;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.hive.ql.exec.vector.VectorizedExpressions;
import org.apache.hadoop.hive.ql.exec.vector.expressions.StringLength;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;

/**
 * UDFLength.
 *
 */
@Description(name = "length",
    value = "_FUNC_(str | binary) - Returns the length of str or number of bytes in binary data",
    extended = "Example:\n"
    + "  > SELECT _FUNC_('Facebook') FROM src LIMIT 1;\n" + "  8")
@VectorizedExpressions({StringLength.class})
public class UDFLength extends UDF {
  private final IntWritable result = new IntWritable();

  public IntWritable evaluate(Text s) {
    if (s == null) {
      return null;
    }

    byte[] data = s.getBytes();
    int len = 0;
    for (int i = 0; i < s.getLength(); i++) {
      if (GenericUDFUtils.isUtfStartByte(data[i])) {
        len++;
      }
    }

    result.set(len);
    return result;
  }

  public IntWritable evaluate(BytesWritable bw){
    if (bw == null){
      return null;

}
    result.set(bw.getLength());
    return result;
  }
}
The implementation of an UDF extends the UDF class and overwrites the evaluate method with its own functionality. UDAFs are a little more complex and require a basic understanding of the map-reduce framework. 

Comments

Post a Comment

Popular posts from this blog

Hive Tutorial 31 : Analytic Functions

Image
Hadoop Hive analytic functions compute an aggregate value that is based on a group of rows. A Hadoop Hive HQL analytic function works on the group of rows and ignores the NULL in the data if you specify Hadoop Hive COUNT Analytic Function Returns number of rows in query or group of rows. Syntax: COUNT(column reference | value expression | *) over(window_spec) For Example; select pat_id, dept_id, count(*) over (partition by dept_id order by dept_id asc) as pat_cnt from patient; at_id dept_id pat_cnt 6 111 4 2 111 4 5 111 4 1 111 4 4 222 3 5 222 3 3 222 3 7 333 1 8 444 1 Hadoop Hive SUM Analytic Function Just like count function, sum Hive analytic function is used to compute the sum of columns or expression. Sum analytic function is used to compute the sum of all rows of table or rows within the groups. Syntax: SUM(column | expression) OVER( window_spec) For example: Calculate sum insured amount of all patients within each department...
Read more

Hive Tutorial 37 : Performance Tuning

Performance plays key role in big data related projects as they deals which huge amount of data. So when you are using Hive if you keep few things in mind then we can see dramatic change in the performance.  Performance tuning in hive: Partitions Bucketing File formats Compression Sampling Tez Vectorization Parallel execution CBO Partitions : The concept of partitioning in Hive is very similar to what we have in RDBMS. A table can be partitioned by one or more keys. This will determine how the data will be stored in the table. For example, if a table has two columns, id, name and age; and is partitioned by age, all the rows having same age will be stored together. So when we try to query based on age range, then hive will retrieve the data by going into particular folders instead of parsing through whole data. /hdfs/user/tablename/age/10 /hdfs/user/tablename/age/11 Bucketing : Bucketing is more efficient for sampling,data will be segre...
Read more