当前位置:Gxlcms > mysql > GPS-GraphProcessingSystem修改源码经验总结(四)

GPS-GraphProcessingSystem修改源码经验总结(四)

时间:2021-07-01 10:21:17 帮助过:28人阅读

本人原创,转载请注明出处!欢迎大家加入 Giraph 技术交流群 : 228591158 本文目的:在修改GPS源码后,详细描述如何编译和分发到各Worker节点上。下面以Graph Coloring 算法为例进行讲解,本文基于GPS的前三篇文章。 1. 首先在Master(test150)上修改源码。G

本人原创,转载请注明出处!欢迎大家加入Giraph 技术交流群: 228591158

本文目的:在修改GPS源码后,详细描述如何编译和分发到各Worker节点上。下面以Graph Coloring 算法为例进行讲解,本文基于GPS的前三篇文章。

1. 首先在Master(test150)上修改源码。Graph Coloring算法源码路径:gps.examples.coloring包,主要修改ColoringVertex.java类。该算法在Section(MIS_1)阶段是按照顶点的出度大小概率性的选择UNDECIDED状态的顶点,源码如下:

if (ColoringVertexType.NOT_IN_SET == value.type
	|| ColoringVertexType.IN_SET == value.type) {
	return;
}
double probability = getNeighborsSize() > 0 ? 1.0 /
    ((double) 2*value.numRemainingNeighbors) : 1;
if (Math.random() <= probability) {
	value.type = ColoringVertexType.SELECTED_AS_POSSIBLE_IN_SET;
	if (value.numRemainingNeighbors > 0) {
	ColoringMessage newSelectedAsPossibleMessage = ColoringMessage
			.newNeighborSelectedAsPossibleMessage(getId());
		for (int neighborId : getNeighborIds()) {
			if (neighborId >= 0) {
				sendMessage(neighborId, newSelectedAsPossibleMessage);
			}
		}
	}
}
下面修改为:选择所有的UNDECIDED状态的顶点,而非概率性选择。修改后源码如下:
if (ColoringVertexType.NOT_IN_SET == value.type
	|| ColoringVertexType.IN_SET == value.type) {
	return;
}
//double probability = getNeighborsSize() > 0 ? 1.0 /
//   ((double) 2*value.numRemainingNeighbors) : 1;
//if (Math.random() <= probability) {
//	value.type = ColoringVertexType.SELECTED_AS_POSSIBLE_IN_SET;
	if (value.numRemainingNeighbors > 0) {
	ColoringMessage newSelectedAsPossibleMessage = ColoringMessage
			.newNeighborSelectedAsPossibleMessage(getId());
		for (int neighborId : getNeighborIds()) {
			if (neighborId >= 0) {
				sendMessage(neighborId, newSelectedAsPossibleMessage);
			}
		}
	}
//}
2. 因为之前已编译过GPS源码,并且把Jar包等文件已分发到集群中各个Worker节点上,所以此步骤是删除之前的文件。删除Master节点trunk目录下的:gps-0.0.1-slave.tar.gz 、gps_node_runner.jar和classes文件夹,删除Worker节点trunk目录下的 conf 、gps-0.0.1-slave.tar.gz 、gps_node_runner.jar 、libs 、scripts,logs文件夹保留。

3. 参考 GPS-Graph Processing System集群安装笔记(一),重新编译和分发Jar包等文件。

下面附上我的脚本,因中间使用了我自己的脚本,故不可直接使用,但是可以参考。脚本所在目录:/home/gougou/GPS/trunk。

cd /home/gougou/GPS/trunk
# delete master files
rm -rf gps_node_runner.jar
rm -rf classes
rm -rf  gps-0.0.1-slave.tar.gz

# delete worker files. the Shell writed by myself.
cd /home/gougou/ShellUtils
./deleteDirectory.sh /home/gougou/GPS/trunk/conf
./deleteDirectory.sh /home/gougou/GPS/trunk/gps-0.0.1-slave.tar.gz
./deleteDirectory.sh /home/gougou/GPS/trunk/gps_node_runner.jar
./deleteDirectory.sh /home/gougou/GPS/trunk/libs
./deleteDirectory.sh /home/gougou/GPS/trunk/scripts/

# compile GPS source code
cd /home/gougou/GPS/trunk
cd local-master-scripts
# generate gps_node_runner.jar and classes under trunk directory
./make_gps_node_runner_jar.sh
# generate  gps-0.0.1-slave.tar.gz under trunk directory
./make_gps_tar_gz.sh

cd ../master-scripts
cp slaves temp
cp slaves-12 slaves
./copy_and_untar_gps_tar_to_slaves.sh 12
mv temp slaves
4. 在trunk/master-scripts目录下,运行Graph Coloring 算法,命令如下:
./start_gps_nodes.sh 2 GC-Test5-1 \
"-ifs /user/gougou/GC-Test5/gc-5.txt \
-hcf /home/gougou/hadoop-1.0.3/conf/core-site.xml \
-jc gps.examples.coloring.JobConfiguration \
-mcfg /machine-configs/test_machine_config_2.cfg \
-log4jconfig /home/gougou/GPS/trunk/conf/log4j.config"
完!

人气教程排行