Hive学习和配置Mysql

1.Hive简介 起源自facebook由Jeff Hammerbacher领导的团队 构建在Hadoop上的数据仓库框架 设计目的是让SQL技能良好，但Java技能较弱的分析师可以查询海量数据 2008年facebook把hive项目贡献给Apache Hive的组件与体系架构 用户接口：shell, thrift, web等 Th

1.Hive简介

 起源自facebook由Jeff Hammerbacher领导的团队
 构建在Hadoop上的数据仓库框架
 设计目的是让SQL技能良好，但Java技能较弱的分析师可以查询海量数据
 2008年facebook把hive项目贡献给Apache

Hive的组件与体系架构

 用户接口：shell, thrift, web等
 Thrift服务器
 元数据库“Derby, Mysql等
 解析器
 Hadoop

Hive安装模式

 内嵌模式：元数据保持在内嵌的Derby模式，只允许一个会话连接(默认)
 本地独立模式：在本地安装Mysql，把元数据放到Mysql内
 远程模式：元数据放置在远程的Mysql数据库

2.配置Mysql

1:copy mysql-connector-java-5.1.6-bin.jar到$HIVE_HOME/lib

[jifeng@jifeng02 hadoop]$ ls
7287OS_Code              hadoop-1.2.1.tar.gz      hive-0.12.0-bin                     tmp
hadoop-1.2.1             hadoop-2.4.1-src.tar.gz  hive-0.12.0-bin.tar.gz
hadoop-1.2.1-bin.tar.gz  hadoop-2.4.1.tar.gz      mysql-connector-java-5.1.6-bin.jar
[jifeng@jifeng02 hadoop]$ cp mysql-connector-java-5.1.6-bin.jar hive-0.12.0-bin/lib

  javax.jdo.option.ConnectionURL
  jdbc:mysql://jifengsql:3306/hive?createDatabaseIfNotExist=true
  JDBC connect string for a JDBC metastore



  javax.jdo.option.ConnectionDriverName
  com.mysql.jdbc.Driver
  Driver class name for a JDBC metastore



  javax.jdo.PersistenceManagerFactoryClass
  org.datanucleus.api.jdo.JDOPersistenceManagerFactory
  class implementing the jdo persistence



  javax.jdo.option.DetachAllOnCommit
  true
  detaches all objects from session so that they can be used after transaction is committed



  javax.jdo.option.NonTransactionalRead
  true
  reads outside of transactions



  javax.jdo.option.ConnectionUserName
  dss
  username to use against metastore database



  javax.jdo.option.ConnectionPassword
  jifeng
  password to use against metastore database

[jifeng@jifeng02 hive-0.12.0-bin]$ hive

Logging initialized using configuration in jar:file:/home/jifeng/hadoop/hive-0.12.0-bin/lib/hive-common-0.12.0.jar!/hive-log4j.properties
hive> show tables;
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
hive> quit;

把 mysql-connector-java-5.1.6-bin.jar 替换成mysql-connector-java-5.1.10-bin.jar还是不行。

检查 mysql发现连接不上，换个虚拟机上mysql

[dss@localhost ~]$ mysql -u root -p
Enter password: 
--root 登陆mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 70
Server version: 5.6.16 MySQL Community Server (GPL)


Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.


Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.


Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.


mysql> create database hive;
Query OK, 1 row affected (0.01 sec) --创建hive库


mysql> GRANT all ON hive.* TO dss@'%' IDENTIFIED BY 'abc123';
Query OK, 0 rows affected (0.03 sec)--给dss用户对hive库授权


mysql> flush privileges;
Query OK, 0 rows affected (0.02 sec)--刷新系统权限表


mysql> set globalbinlog_format='MIXED'; 
ERROR 1193 (HY000): Unknown system variable 'globalbinlog_format'
mysql> alter database hive character set latin1 ;
Query OK, 1 row affected (0.00 sec)--更新字符集

再次启动

[jifeng@jifeng02 hive-0.12.0-bin]$ hive

Logging initialized using configuration in jar:file:/home/jifeng/hadoop/hive-0.12.0-bin/lib/hive-common-0.12.0.jar!/hive-log4j.properties
hive> show tables;
OK
Time taken: 6.273 seconds
hive> 

4.Hive的运行模式即任务的执行环境

1启动hive 命令行模式：

1：直接输入#hive的执行程序，

2：或者输入 #hive --service cli

l 分为本地与集群两种

我们可以通过mapred.job.tracker 来指明

设置方式：

hive >

2.hive验证启动的方法

l 1、hive web界面的(端口号9999) 启动方式

#hive --service hwi

[jifeng@jifeng02 hive-0.12.0-bin]$ hive --service cli

Logging initialized using configuration in jar:file:/home/jifeng/hadoop/hive-0.12.0-bin/lib/hive-common-0.12.0.jar!/hive-log4j.properties
hive> quit;
[jifeng@jifeng02 hive-0.12.0-bin]$ hive --service hwi 
15/08/17 15:17:10 INFO hwi.HWIServer: HWI is starting up
15/08/17 15:17:10 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
15/08/17 15:17:10 INFO mortbay.log: jetty-6.1.26
15/08/17 15:17:10 INFO mortbay.log: Extract /home/jifeng/hadoop/hive-0.12.0-bin/lib/hive-hwi-0.12.0.war to /tmp/Jetty_0_0_0_0_9999_hive.hwi.0.12.0.war__hwi__ow27i/webapp
15/08/17 15:17:11 INFO mortbay.log: Started SocketConnector@0.0.0.0:9999

用于通过浏览器来访问hive

http://jifeng02:9999/hwi/

l 2、hive 远程服务(端口号10000) 启动方式

#hive --service hiveserver