Create Local Cloudera Parcels Repo to Save Your ASS

If you want to install Hadoop cluster via Cloudera Manager in CHINA, the network to cloudera parels is toooooo slow. You must create your local parcel repo to save your ass, thanks to the DAMN GFW!!

 

Follow this page from cloudera:

• Download parcels:    

            Link to CDH4：http://archive.cloudera.com/cdh4/parcels/                                                                                 
            Link to IMPALA：http://archive.cloudera.com/impala/parcels/

• Verify the location of the local parcel repository on your Cloudera Manager server: Go to the Administration page -> Properties tab -> Parcels category. You can change the local repository path in the Local Parcel Repository Path property. By default it is /opt/cloudera/parcel-repo.
• Download the .parcel file for your operating system: (el5 or el6 for Red Hat 5 or 6, lucid or precise for Ubuntu and so on) and place it into the local parcel repository on your Cloudera Manager server. In my case it's CDH-4.3.0-1.cdh4.3.0.p0.22-el6.parcel and IMPALA-1.0.1-1.p0.431-el6.parcel
• Open the manifest.json file in the same directory as the .parcel file you just copied. Find the section of the manifest that corresponds to the parcel you downloaded: In my case, I'm running RHEL 6 and copied the parcel file CDH-4.2.0-1.cdh4.2.0.p0.10-el6.parcel, then look for the section:

    {
        "parcelName": "CDH-4.3.0-1.cdh4.3.0.p0.22-el6.parcel",
        "components": [
          { "name":     "flume-ng",
            "version":  "1.3.0-cdh4.3.0",
            "pkg_version":  "1.3.0+159" 
          }
          ,{ "name":     "hadoop-0.20-mapreduce",
            "version":  "2.0.0-cdh4.3.0",
            "pkg_version":  "2.0.0+1357" 
          }
          ,{ "name":     "hadoop-hdfs",
            "version":  "2.0.0-cdh4.3.0",
            "pkg_version":  "2.0.0+1357" 
          }
          ,{ "name":     "hadoop-httpfs",
            "version":  "2.0.0-cdh4.3.0",
            "pkg_version":  "2.0.0+1357" 
          }
          ,{ "name":     "hadoop-mapreduce",
            "version":  "2.0.0-cdh4.3.0",
            "pkg_version":  "2.0.0+1357" 
          }
          ,{ "name":     "hadoop-yarn",
            "version":  "2.0.0-cdh4.3.0",
            "pkg_version":  "2.0.0+1357" 
          }
          ,{ "name":     "hadoop",
            "version":  "2.0.0-cdh4.3.0",
            "pkg_version":  "2.0.0+1357" 
          }
          ,{ "name":     "hbase",
            "version":  "0.94.6-cdh4.3.0",
            "pkg_version":  "0.94.6+96" 
          }
          ,{ "name":     "hcatalog",
            "version":  "0.5.0-cdh4.3.0",
            "pkg_version":  "0.5.0+9" 
          }
          ,{ "name":     "hive",
            "version":  "0.10.0-cdh4.3.0",
            "pkg_version":  "0.10.0+121" 
          }
          ,{ "name":     "mahout",
            "version":  "0.7-cdh4.3.0",
            "pkg_version":  "0.7+16" 
          }
          ,{ "name":     "oozie",
            "version":  "3.3.2-cdh4.3.0",
            "pkg_version":  "3.3.2+49" 
          }
          ,{ "name":     "pig",
            "version":  "0.11.0-cdh4.3.0",
            "pkg_version":  "0.11.0+28" 
          }
          ,{ "name":     "sqoop",
            "version":  "1.4.3-cdh4.3.0",
            "pkg_version":  "1.4.3+34" 
          }
          ,{ "name":     "sqoop2",
            "version":  "1.99.1-cdh4.3.0",
            "pkg_version":  "1.99.1+115" 
          }
          ,{ "name":     "whirr",
            "version":  "0.8.2-cdh4.3.0",
            "pkg_version":  "0.8.2+10" 
          }
          ,{ "name":     "zookeeper",
            "version":  "3.4.5-cdh4.3.0",
            "pkg_version":  "3.4.5+19" 
          }
          ,{ "name":     "hue",
            "version":  "2.3.0-cdh4.3.0",
            "pkg_version":  "2.3.0+136" 
          }
        ],
        "hash": "df5cc61b2d257aaf625341f709a4f8e09754038a"
      }

   

    {
        "parcelName": "IMPALA-1.0.1-1.p0.431-el6.parcel",
        "components": [
          { "name":     "impala",
            "version":  "1.0.1-SNAPSHOT",
            "pkg_version":  "1.0.1"
          }
        ],
        "hash": "992467f2e54bd394cbdd3f4ed97b6e9bead60ff0"
      }

   
• Create a text file whose name is <parcel file name> .sha (e.g. CDH-4.2.0-1.cdh4.2.0.p0.10-el6.parcel.sha) and copy the hash code into it. In my case, I created file CDH-4.3.0-1.cdh4.3.0.p0.22-el6.parcel.sha and IMPALA-1.0.1-1.p0.431-el6.parcel.sha, then I copied the hash value into them respectively:

  # cat > CDH-4.3.0-1.cdh4.3.0.p0.22-el6.parcel.sha
  df5cc61b2d257aaf625341f709a4f8e09754038a
  ^C

   

  # cat > IMPALA-1.0.1-1.p0.431-el6.parcel.sha
  992467f2e54bd394cbdd3f4ed97b6e9bead60ff0
  ^C

   Place both of files into your local parcel repository, /opt/cloudera/parcel-repo by default, depending you settings in the above step.
• Once these files are in place, Cloudera Manager will pick up the parcel and it will appear on the Hosts > Parcels page. Note that how quickly this occurs depends on the Parcel Update Frequency setting, set by default to 1 hour. You can change this on the Administration page -> Properties tab -> Parcels category.

评论

1 楼 NIghtmare28 2014-07-17  

  太好用了, 谢谢