当前位置:Gxlcms > 数据库问题 > MongoDB 自动分片 auto sharding

MongoDB 自动分片 auto sharding

时间:2021-07-01 10:21:17 帮助过:3人阅读

/home/conan/dbs ~ mkdir config1 config2 config3 shard1 shard2 shard3 ~/dbs$ ls -l drwxrwxr-x 3 conan conan 4096 May 31 11:27 config1 drwxrwxr-x 3 conan conan 4096 May 31 11:27 config2 drwxrwxr-x 3 conan conan 4096 May 31 11:27 config3 drwxrwxr-x 3 conan conan 4096 May 31 11:28 shard1 drwxrwxr-x 3 conan conan 4096 May 31 11:29 shard2 drwxrwxr-x 3 conan conan 4096 May 31 11:29 shard3

2. 启动shard节点

技术分享

启动config节点

mongod --dbpath config1 --port 20001 --fork --logpath config1.log
mongod --dbpath config2 --port 20002 --fork --logpath config2.log
mongod --dbpath config3 --port 20003 --fork --logpath config3.log

启动mongos节点

mongos --configdb localhost:20001,localhost:20002,localhost:20003 --port 30001 --fork --logpath mongos1.log
mongos --configdb localhost:20001,localhost:20002,localhost:20003 --port 30002 --fork --logpath mongos2.log

启动shard节点

mongod --dbpath shard1 --port 10001 --fork --logpath shard1.log
mongod --dbpath shard2 --port 10002 --fork --logpath shard2.log
mongod --dbpath shard3 --port 10003 --fork --logpath shard3.log

查看端口

~ netstat -nlt

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State      
tcp        0      0 0.0.0.0:10001               0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:30001               0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:10002               0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:30002               0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:10003               0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:21                  0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:20001               0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:20002               0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:20003               0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:27017             0.0.0.0:*                   LISTEN      
tcp        0      0 :::22                       :::*                        LISTEN      
tcp        0      0 ::1:631                     :::*                        LISTEN      
tcp        0      0 ::1:25                      :::*                        LISTEN  

3. 配置shard节点

连接mongos1,在mongos中添加分片:

# mongo localhost:30001/admin
MongoDB shell version: 3.0.5
connecting to: localhost:30001/admin
Server has startup warnings: 
2015-08-12T23:37:56.997-0700 I CONTROL  ** WARNING: You are running this process as the root user, which is not recommended.
2015-08-12T23:37:56.997-0700 I CONTROL  
mongos> db.runCommand({addshard : "localhost:10001", allowLocal : true})
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> db.runCommand({addshard : "localhost:10002", allowLocal : true})
{ "shardAdded" : "shard0001", "ok" : 1 }
mongos> db.runCommand({addshard : "localhost:10003", allowLocal : true})
{ "shardAdded" : "shard0002", "ok" : 1 }

 

查看分片信息

mongos> db.runCommand({listshards:1})
{
        "shards" : [
                {
                        "_id" : "shard0000",
                        "host" : "localhost:10001"
                },
                {
                        "_id" : "shard0001",
                        "host" : "localhost:10002"
                },
                {
                        "_id" : "shard0002",
                        "host" : "localhost:10003"
                }
        ],
        "ok" : 1
}

启用数据库分片:fensme

 

mongos> db.runCommand({"enablesharding" : "fensme"})
{ "ok" : 1 }

注:一旦enable了个数据库,mongos将会把数据库里的不同数据集放在不同的分片上。只有数据集也被分片,否则一个数据集的所有数据将放在一个分片上。

启用数据集分片:fensme.users

db.runCommand({"shardcollection" : "fensme.users", "key" : {"_id" : 1,"uid":1}})
{ "collectionsharded" : "fensme.users", "ok" : 1 }

查看分片状态

 

mongos> db.printShardingStatus()
--- Sharding Status --- 
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("55cc3b46396d4ce739e9eacc")
}
  shards:
        {  "_id" : "shard0000",  "host" : "localhost:10001" }
        {  "_id" : "shard0001",  "host" : "localhost:10002" }
        {  "_id" : "shard0002",  "host" : "localhost:10003" }
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                No recent migrations
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "fensme",  "partitioned" : true,  "primary" : "shard0000" }
                fensme.users
                        shard key: { "_id" : 1, "uid" : 1 }
                        chunks:
                                shard0000       1
                        { "_id" : { "$minKey" : 1 }, "uid" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 }, "uid" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0) 

fensme数据库是支持shard的,主shard是shard0000,对应host是localhost:10001

再查看config数据库

mongos> use config
switched to db config
mongos> show collection
actionlog
changelog
chunks
collections
databases
lockpings
locks
mongos
settings
shards
system.indexes
tags
version
mongos> db.shards.find()
{ "_id" : "shard0000", "host" : "localhost:10001" }
{ "_id" : "shard0001", "host" : "localhost:10002" }
{ "_id" : "shard0002", "host" : "localhost:10003" }
mongos> db.chunks.find()
{ "_id" : "fensme.users-_id_MinKeyuid_MinKey", "lastmod" : Timestamp(1, 0), "lastmodEpoch" : ObjectId("55cc4b46396d4ce739e9edcf"), "ns" : "fensme.users", "min" : { "_id" : { "$minKey" : 1 }, "uid" : { "$minKey" : 1 } }, "max" : { "_id" : { "$maxKey" : 1 }, "uid" : { "$maxKey" : 1 } }, "shard" : "shard0000" }

shards配置成功,分片信息都正确

4. 插入数据分片实验

向fensme.users插入数据, 批量插入10w条记录

mongos> use fensme
switched to db fensme
mongos>  for(var i=0; i<10000; i++){ 
... db.users.insert({_id:i*1597,uid:i});
... }
WriteResult({ "nInserted" : 1 })
mongos> db.users.find()
{ "_id" : 0, "uid" : 0 }
{ "_id" : 1597, "uid" : 1 }
{ "_id" : 33537, "uid" : 21 }
{ "_id" : 3194, "uid" : 2 }
{ "_id" : 35134, "uid" : 22 }
{ "_id" : 4791, "uid" : 3 }
{ "_id" : 36731, "uid" : 23 }
{ "_id" : 6388, "uid" : 4 }
{ "_id" : 38328, "uid" : 24 }
{ "_id" : 7985, "uid" : 5 }
{ "_id" : 39925, "uid" : 25 }
{ "_id" : 9582, "uid" : 6 }
{ "_id" : 41522, "uid" : 26 }
{ "_id" : 11179, "uid" : 7 }
{ "_id" : 43119, "uid" : 27 }
{ "_id" : 12776, "uid" : 8 }
{ "_id" : 44716, "uid" : 28 }
{ "_id" : 14373, "uid" : 9 }
{ "_id" : 46313, "uid" : 29 }
{ "_id" : 15970, "uid" : 10 }
Type "it" for more

查看数据分片存储信息

 

mongos> db.users.stats()
{
        "sharded" : true,
        "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
        "userFlags" : 1,
        "capped" : false,
        "ns" : "fensme.users",
        "count" : 10000,
        "numExtents" : 6,
        "size" : 480048,
        "storageSize" : 712704,
        "totalIndexSize" : 711312,
        "indexSizes" : {
                "_id_" : 302512,
                "_id_1_uid_1" : 408800
        },
        "avgObjSize" : 48.0048,
        "nindexes" : 2,
        "nchunks" : 3,
        "shards" : {
                "shard0000" : {
                        "ns" : "fensme.users",
                        "count" : 9979,
                        "size" : 479040,
                        "avgObjSize" : 48,
                        "numExtents" : 4,
                        "storageSize" : 696320,
                        "lastExtentSize" : 524288,
                        "paddingFactor" : 1,
                        "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
                        "userFlags" : 1,
                        "capped" : false,
                        "nindexes" : 2,
                        "totalIndexSize" : 678608,
                        "indexSizes" : {
                                "_id_" : 286160,
                                "_id_1_uid_1" : 392448
                        },
                        "ok" : 1
                },
                "shard0001" : {
                        "ns" : "fensme.users",
                        "count" : 1,
                        "size" : 48,
                        "avgObjSize" : 48,
                        "numExtents" : 1,
                        "storageSize" : 8192,
                        "lastExtentSize" : 8192,
                        "paddingFactor" : 1,
                        "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
                        "userFlags" : 1,
                        "capped" : false,
                        "nindexes" : 2,
                        "totalIndexSize" : 16352,
                        "indexSizes" : {
                                "_id_" : 8176,
                                "_id_1_uid_1" : 8176
                        },
                        "ok" : 1
                },
                "shard0002" : {
                        "ns" : "fensme.users",
                        "count" : 20,
                        "size" : 960,
                        "avgObjSize" : 48,
                        "numExtents" : 1,
                        "storageSize" : 8192,
                        "lastExtentSize" : 8192,
                        "paddingFactor" : 1,
                        "paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
                        "userFlags" : 1,
                        "capped" : false,
                        "nindexes" : 2,
                        "totalIndexSize" : 16352,
                        "indexSizes" : {
                                "_id_" : 8176,
                                "_id_1_uid_1" : 8176
                        },
                        "ok" : 1
                }
        },
        "ok" : 1
}

查看users数据集信息,shard0000上面9979条,shard0001上面1条,shard0002上面20条。分片很不均匀。

分别连接到shard1,shard2,shard3查看数据分片存储的情况
连接shard1

# mongo localhost:10001
MongoDB shell version: 3.0.5
connecting to: localhost:10001/test> show dbsshow dbs
fensme  0.078GB
local   0.078GB
> use fensme
switched to db fensme
> show collections
system.indexes
users
> db.users.count()
9979

连接shard2, 1条记录

 

# mongo localhost:10002
MongoDB shell version: 3.0.5
connecting to: localhost:10002/test> use fensme
switched to db fensme
> db.users.count()
1

连接shard3,20

 

# mongo localhost:10003
MongoDB shell version: 3.0.5
connecting to: localhost:10003/test> use fensme
switched to db fensme
> db.users.count()
20

注:分片数据分配不均匀,应该重新规化分片的key。

5. 删除主分片

移除Primary分片shard1,localhost:10001

mongos> db.runCommand({"removeshard":"localhost:10001"})
{
        "msg" : "draining started successfully",
        "state" : "started",
        "shard" : "shard0000",
        "note" : "you need to drop or movePrimary these databases",
        "dbsToMove" : [
                "fensme"
        ],
        "ok" : 1
}

提示shard0000是主分片,要手动迁移主分片。

查看分片状态

mongos> db.runCommand({listshards:1})
{
        "shards" : [
                {
                        "_id" : "shard0000",
                        "host" : "localhost:10001",
                        "draining" : true
                },
                {
                        "_id" : "shard0001",
                        "host" : "localhost:10002"
                },
                {
                        "_id" : "shard0002",
                        "host" : "localhost:10003"
                }
        ],
        "ok" : 1
}

draining为正在迁移过程中。。。。

 

mongos> db.printShardingStatus()
--- Sharding Status --- 
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("55cc3b46396d4ce739e9eacc")
}
  shards:
        {  "_id" : "shard0000",  "host" : "localhost:10001",  "draining" : true }
        {  "_id" : "shard0001",  "host" : "localhost:10002" }
        {  "_id" : "shard0002",  "host" : 
                        
                    

人气教程排行