hadoopmapreduce多表关联

hadoop mapreduce多表关联 假设有如下两个文件，一个是表是公司和地址的序号的对应，一个表是地址的序号和地址的名称的对应。 表1： [plain] A:Beijing Red Star 1 A:Shenzhen Thunder 3 A:Guangzhou Honda 2 A:Beijing Rising 1 A:Guangzhou Development Ba

hadoop mapreduce多表关联

假设有如下两个文件，一个是表是公司和地址的序号的对应，一个表是地址的序号和地址的名称的对应。

表1：

[plain]

A:Beijing Red Star 1

A:Shenzhen Thunder 3

A:Guangzhou Honda 2

A:Beijing Rising 1

A:Guangzhou Development Bank 2

A:Tencent 3

A:Back of Beijing 1

表2：

[plain]

B:1 Beijing

B:2 Guangzhou

B:3 Shenzhen

B:4 Xian

mapreduce如下：

[plain]

private static final Text typeA = new Text("A:");

private static final Text typeB = new Text("B:");

private static Log log = LogFactory.getLog(MTJoin.class);

public static class Map extends Mapper {

public void map(Object key, Text value, Context context)

throws IOException, InterruptedException {

String valueStr = value.toString();

String type = valueStr.substring(0, 2);

String content = valueStr.substring(2);

log.info(content);

if(type.equals("A:"))

{

String[] contentArray = content.split("\t");

String city = contentArray[0];

String address = contentArray[1];

MapWritable map = new MapWritable();

map.put(typeA, new Text(city));

context.write(new Text(address), map);

}

else if(type.equals("B:"))

{

String[] contentArray = content.split("\t");

String adrNum = contentArray[0];

String adrName = contentArray[1];

MapWritable map = new MapWritable();

map.put(typeB, new Text(adrName));

context.write(new Text(adrNum), map);

}

}

}

public static class Reduce extends Reducer {

public void reduce(Text key, Iterable values, Context context)

throws IOException, InterruptedException {

Iterator it = values.iterator();

List cityList = new ArrayList();

List adrList = new ArrayList();

while(it.hasNext())

{

MapWritable map = it.next();

if(map.containsKey(typeA))

{

cityList.add((Text)map.get(typeA));

}

else if(map.containsKey(typeB))

{

adrList.add((Text)map.get(typeB));

}

}

for(int i = 0; i < cityList.size(); i++)

{

for(int j = 0; j < adrList.size(); j++)

{

context.write(cityList.get(i), adrList.get(j));

}

}

}

}

原理很简单，map的出口，以地址的序号作为key，然后出来的时候，公司名称放一个list，地址的名称放一个list，两个list的内容作笛卡儿积，就得到了结果。

输出如下：

[plain]

Beijing Red Star Beijing

Beijing Rising Beijing

Back of Beijing Beijing

Guangzhou Honda Guangzhou

Guangzhou Development Bank Guangzhou

Shenzhen Thunder Shenzhen

Tencent Shenzhen