MongoDB Oplogs 到底都记录了什么与智者老冯

开头还是介绍一下群，如果感兴趣PolarDB ,MongoDB ,MySQL ,PostgreSQL ,Redis, Oceanbase, Sql Server等有问题，有需求都可以加群群内有各大数据库行业大咖，CTO，可以解决你的问题。加群请联系 liuaustin3 ，（共1800人左右 1 + 2 + 3 + 4 +5） 4群（300+）,另欢迎 OpenGauss 的技术人员加入。

最近董宇辉的事情，让我意识到，如果光一味的搞技术，那么对于人生和生活是不完整的，后序在撰写一些技术文章的时候，会带有一些对于当前热点事件的一些感触和反思，也希望能找到一些有同样想法的人，终究人生的道路是孤独的，如果在孤独中能找到一些人能对你有一些共鸣，那是一件幸福的事情。

这里先说技术，后说感触：

MongoDB 本身是一个分成收到欢迎的 NOSQL数据库，其中oplogs是MongoDB 中最重要的日志，他提供了强大的复制和同步的能力，由此mongodb 才能有高可用，复制集合，数据一致性，分片等多种能力的支持，这些都与oplogs 有强相关。

在这里我们先说说oplogs 的数据中的格式

repl:PRIMARY> show collections;
oplog.rs
replset.election
replset.minvalid
replset.oplogTruncateAfterPoint
startup_log
system.profile
system.replset
system.rollback.id
repl:PRIMARY> db.oplog.rs.findOne();
{
 "ts" : Timestamp(1639190792, 1),
 "h" : NumberLong(0),
 "v" : 2,
 "op" : "n",
 "ns" : "",
 "wall" : ISODate("2021-12-11T02:46:32.518Z"),
 "o" : {
  "msg" : "initiating set"
 }
}
repl:PRIMARY> db.oplog.rs.find();
{ "ts" : Timestamp(1639190792, 1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:46:32.518Z"), "o" : { "msg" : "initiating set" } }
{ "ts" : Timestamp(1639190803, 2), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "c", "ns" : "config.$cmd", "ui" : UUID("c3c70f0e-028e-4db8-85b4-79e8b65eccc9"), "wall" : ISODate("2021-12-11T02:46:43.802Z"), "o" : { "create" : "transactions", "idIndex" : { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "config.transactions" } } }
{ "ts" : Timestamp(1639190803, 3), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "c", "ns" : "config.$cmd", "ui" : UUID("b2fb5908-b21b-4ed5-9f2d-218396875b19"), "wall" : ISODate("2021-12-11T02:46:43.806Z"), "o" : { "create" : "image_collection", "idIndex" : { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "config.image_collection" } } }
{ "ts" : Timestamp(1639190803, 4), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:46:43.806Z"), "o" : { "msg" : "new primary" } }
{ "ts" : Timestamp(1639190803, 5), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "c", "ns" : "admin.$cmd", "ui" : UUID("4a7fa891-c1a9-49a6-abc4-d94693030358"), "wall" : ISODate("2021-12-11T02:46:43.810Z"), "o" : { "create" : "system.keys", "idIndex" : { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "admin.system.keys" } } }
{ "ts" : Timestamp(1639190803, 6), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "i", "ns" : "admin.system.keys", "ui" : UUID("4a7fa891-c1a9-49a6-abc4-d94693030358"), "wall" : ISODate("2021-12-11T02:46:43.811Z"), "o" : { "_id" : NumberLong("7040270890788978692"), "purpose" : "HMAC", "key" : BinData(0,"aIg6SYcKvt6nEuNtwy/ZljTSHB8="), "expiresAt" : Timestamp(1646966803, 0) } }
{ "ts" : Timestamp(1639190804, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "i", "ns" : "admin.system.keys", "ui" : UUID("4a7fa891-c1a9-49a6-abc4-d94693030358"), "wall" : ISODate("2021-12-11T02:46:44.561Z"), "o" : { "_id" : NumberLong("7040270890788978693"), "purpose" : "HMAC", "key" : BinData(0,"1PeuaQ4B1+eGmmlBuaGU+NfTcpA="), "expiresAt" : Timestamp(1654742803, 0) } }
{ "ts" : Timestamp(1639190823, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:47:03.807Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190833, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:47:13.808Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190843, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:47:23.809Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190853, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:47:33.809Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190863, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:47:43.810Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190873, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:47:53.811Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190883, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:48:03.812Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190893, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:48:13.812Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190903, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:48:23.813Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190913, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:48:33.813Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190923, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:48:43.815Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190933, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:48:53.816Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639190943, 1), "t" : NumberLong(1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-11T02:49:03.817Z"), "o" : { "msg" : "periodic noop" } }
Type "it" for more

首先我们可以打开oplogs并且可以通过语句来对数据进行读取，上面的oplogs 中的一些字段是有意义的，我们这里来解释一下。

ts: 时间戳，基于是日志，并且是一个分布式数据库的日志，则每个操作的时间是非常重要的，这里时间戳的产生都要和复制集中的各个主机的时间进行同步，并且产生的时间戳在各个复制集中具有唯一性。这里 ts 64位来表达这个时间 32位时间 + 32位计数累计

t: 这里与mongodb的本身复制集有关，基于复制集采用的raft 协议，则在每次节点进行切换的情况下，这里的值会进行自增，来标记这个节点曾经发生了什么。

v: oplog的版本标识

op：数据库中具体的操作这里有不同的标识，如插入，删除，更新，DDL 空转等标志， i , d , u ,c, n

ns: 操作在那个层面发生如某个 collation 里面

o: 操作中指定的字段 key 与 op 进行配置和，则获得对于操作+字段，则指定这个操作的具体信息

o2: 查询字段，这里只针对 op=u的情况下

b: 这标识在数据插入中，使用的是upsert 如果这个位置是 true，则表明数据是upsert 方式插入的

txnNumber 标识事务的ID 因为MongoDB是支持事务的，所以事务也是有事务号的，也是有顺序和标识的

当然这里还有一些其他的字段，比如prevOpTime, preImageOpTime, postImangeOpTime ,这些字段是针对当前的oplogs 条目与前后与这条记录的有关系的条目关系。

repl:PRIMARY> db.test.insertOne({"author" : "Simon", "text" : "Lemon is sweet", "tags":["mongodb", "python", "pymongo"], "date": ISODate("2023-12-18T04:22:47.243Z") });
{
 "acknowledged" : true,
 "insertedId" : ObjectId("61b78ad32b236dddb7d04847")
}
repl:PRIMARY> db.test.insertOne({"author" : "Simon", "text" : "Lemon is sweet", "tags":["mongodb", "python", "pymongo"], "date": ISODate("2023-12-18T04:22:47.243Z") });
{
 "acknowledged" : true,
 "insertedId" : ObjectId("61b78ad42b236dddb7d04848")
}
repl:PRIMARY> db.test.insertOne({"author" : "Simon", "text" : "Lemon is sweet", "tags":["mongodb", "python", "pymongo"], "date": ISODate("2023-12-18T04:22:47.243Z") });
{
 "acknowledged" : true,
 "insertedId" : ObjectId("61b78ad52b236dddb7d04849")
}
repl:PRIMARY>

这里我们在一个表中插入数据，并尝试在oplogs 中找到这条数据

repl:PRIMARY> db.oplog.rs.find({ "op": "i" }).sort({ $natural: -1 }).limit(1);
{ "ts" : Timestamp(1639418581, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "i", "ns" : "test.test", "ui" : UUID("28a721ca-06b1-4c06-8588-efb0ef4fd21d"), "wall" : ISODate("2021-12-13T18:03:01.880Z"), "o" : { "_id" : ObjectId("61b78ad52b236dddb7d04849"), "author" : "Simon", "text" : "Lemon is sweet", "tags" : [ "mongodb", "python", "pymongo" ], "date" : ISODate("2023-12-18T04:22:47.243Z") } }
repl:PRIMARY>

db.oplog.rs.find({ "op": "i" }).sort({ $natural: -1 }).limit(1);

{ "ts" : Timestamp(1639418581, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "i", "ns" : "test.test", "ui" : UUID("28a721ca-06b1-4c06-8588-efb0ef4fd21d"), "wall" : ISODate("2021-12-13T18:03:01.880Z"), "o" : { "_id" : ObjectId("61b78ad52b236dddb7d04849"), "author" : "Simon", "text" : "Lemon is sweet", "tags" : [ "mongodb", "python", "pymongo" ], "date" : ISODate("2023-12-18T04:22:47.243Z") } }

这里我们可以看出，在oplogs中的确有标记，

"ns" : "test.test"

"o" : { "_id" : ObjectId("61b78ad52b236dddb7d04849"), "author" : "Simon", "text" : "Lemon is sweet", "tags" : [ "mongodb", "python", "pymongo" ], "date" : ISODate("2023-12-18T04:22:47.243Z") } }

这两个部分已经标记了整体的操作的信息，凭借这些完全可以进行数据的复制操作。

repl:PRIMARY> db.oplog.rs.find({ "op": "c" }).sort({ $natural: -1 }).limit(1);
{ "ts" : Timestamp(1639419500, 2), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "c", "ns" : "test.$cmd", "ui" : UUID("28a721ca-06b1-4c06-8588-efb0ef4fd21d"), "wall" : ISODate("2021-12-13T18:18:20.703Z"), "o" : { "createIndexes" : "test", "v" : 2, "key" : { "author" : -1 }, "name" : "author_-1", "background" : true } }
repl:PRIMARY>

从上面的部分我们也可以看到添加索引的DDL 也明确可以在oplogs中存在添加索引的记录。

在我们感性的认知了oplogs后，我们针对oplogs在性能方面进行一些说明

1 关于oplogs 的大小的问题，在设置oplogs空间的大小与这个数据库索要承载的数据库的并发和数据的处理量有关。

2 网络的问题是针对mongodb中尤其是非单机的情况下，性能，数据的一致性，包含在一些情况下，数据的写入的速度等等都有关，所以网络对于mongodb 的重要性不言而喻。

同时我们还有一些对于oplogs 的查询和分析语句

比如我们查询1小时内的oplogs的记录

var since = Math.floor(ISODate("2021-12-13T09:00:00.000Z").getTime() / 1000)

var until = Math.floor(ISODate("2021-12-13T15:00:00.000Z").getTime() / 1000)

db.oplog.rs.find({ "ts" : { "

repl:PRIMARY> var  since = Math.floor(ISODate("2021-12-13T09:00:00.000Z").getTime() / 1000)
repl:PRIMARY> var  until = Math.floor(ISODate("2021-12-13T15:00:00.000Z").getTime() / 1000)
repl:PRIMARY> db.oplog.rs.find({ "ts"  : { "$lt"  : Timestamp(until, 1), "$gt"  : Timestamp(since, 1) } })
{ "ts" : Timestamp(1639388480, 2), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:41:20.144Z"), "o" : { "msg" : "new primary" } }
{ "ts" : Timestamp(1639388490, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:41:30.150Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388500, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:41:40.156Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388510, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:41:50.160Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388520, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:42:00.164Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388530, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:42:10.164Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388540, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:42:20.167Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388550, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:42:30.169Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388560, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:42:40.171Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388570, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:42:50.173Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388580, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:43:00.175Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388590, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:43:10.179Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388600, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:43:20.180Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388610, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:43:30.184Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388620, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:43:40.187Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388630, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:43:50.190Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388640, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:44:00.193Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388650, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:44:10.196Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388660, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:44:20.197Z"), "o" : { "msg" : "periodic noop" } }
{ "ts" : Timestamp(1639388670, 1), "t" : NumberLong(2), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "wall" : ISODate("2021-12-13T09:44:30.198Z"), "o" : { "msg" : "periodic noop" } }
Type "it" for more
repl:PRIMARY>

统计当前oplog中的update 操作

repl:PRIMARY> db.oplog.rs.aggregate([
...         { $match: { "op":"u"  }},
...         { $group: { _id:"$ns",count:{$sum:1}}}
...     ])
{ "_id" : "admin.system.users", "count" : 1 }
{ "_id" : "config.system.sessions", "count" : 12 }
repl:PRIMARY>

最近董宇辉的事情，让我对于冯唐冯老师，非常的佩服，尤其是回顾东方甄选有一期，冯唐老师和董宇辉和俞敏洪直播的视频，进行翻看后，冯唐老师对于人性的把握，对于企业的发展，对于员工和老板之间关系发展，，以及利益的取舍等，都有非常直白的解读，尽管俞敏洪将话讲的非常的“文艺”，而冯唐厉害之处，将常人听不懂的内涵的意思，全部解读出来，并且正大光明的将做这些话说出来，这是很多名人都做不到的。

这里给我一些启示：

1 人是有高低之分，承认人和人就是不一样，这与公平无关，这是现实。

2 人可以善良，但是善良之前，你是否有能力支撑你的善良，善良不等于软弱和无知。

3 可以收敛自己的聪明，但对跨过底线人，必须有聪明去惩罚过线的人让其得到教训或惩罚。

4 做人见好就收，不要赚最后一个铜板，不要将自己推到巅峰，在半山腰下来的未必不是好事。

MongoDB Oplogs 到底都记录了什么 与 智者老冯

MongoDB Oplogs 到底都记录了什么与智者老冯