↓Skip to main content
  1. Posts/

Configuring PDI for Use with Hadoop 1.2.0

·269 words·2 mins·
Software Things
Table of Contents

Purpose #

In this post I have captured my work configuring Pentaho Data Integration (PDI) for use with Hadoop. It is formatted as a tutorial on how to setup PDI 4.4 with Hadoop 1.2.0 for your use.

Prerequisites #

  • Java 1.6 or later (Not the OpenJDK distro as it is not compatible with this version of PDI)
  • Pentaho Data Integration 4.4
  • An up and running Hadoop 1.2.0 Cluster

Configuring up Pentaho #

Pentaho comes preconfigured for use with Hadoop 0.2.0. Which is great…unless you want to use a different version of Hadoop. The supported versions of Hadoop for use with Pentaho are outlined in their support matrix that can be found in the Pentaho InfoCenter. In my case, I was using Hadoop 1.2.0 so I found that it is necessary to create your own Hadoop configuration for Pentaho. I augmented the instructions located in the Pentaho InfoCenter on “Creating a New Hadoop Configuration” with the instructions found in this article. See updated instructions below:

  1. Go into the $PDI_HOME/plugins/pentaho-big-data-plugin/hadoop-configurations directory.

  2. Make a copy of the hadoop-20 folder and rename it to hadoop-120. This folder is the name of your new configuration.

  3. Copy the following JAR files from your Hadoop NameNode into the $PDI_HOME/plugins/pentaho-big-data-plugin/hadoop-configurations/hadoop-120/lib/client directory:

    • commons-codec-1.4.jar
    • commons-configuration-1.6.jar
    • hadoop-core-1.2.0.jar

  4. Remove these files after copying in the updated libraries:

    • commons-codec-1.3.jar
    • hadoop-core-0.20.2.jar

  5. Update theactive.hadoop.configurationproperty, which configures the distribution of Hadoop that PDI will use when communicating with the cluster, in the $PDI_HOME/plugins/pentaho-big-data-plugin/plugin.properties file to look like the following code block:

    active.hadoop.configuration=hadoop-120

Once the steps above have been completed then you can start PDI and begin using the Big Data steps like Hadoop Copy Files and MapReduce.

↑