elasticsearch 和 mongoldb 同步数据时的错误

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

现在注册

已注册用户请登录

• 请不要在回答技术问题时复制粘贴 AI 生成的内容

这是一个创建于 3030 天前的主题，其中的信息可能已经有所发展或是发生改变。

mongo-connector.log 的内容是

OperationFailed: TransportError(404, u'{"_index":"nicovideo","_type":"posts","_id":"27131240","found":false}')
2016-09-10 00:01:36,787 [ERROR] mongo_connector.oplog_manager:324 - Unable to process oplog document {u'h': -4790094769725122799L, u'ts': Timestamp(1473480246, 2), u'o': {u'$set': {u'view_count': 22, u'__v': 5, u'urls': [{u'cookie': u'sm29618346:1473480091:1473480091:0747757f18bebe4a:1', u'vip': False, u'type': u'mp4', u'get_at': datetime.datetime(2016, 9, 10, 4, 4, 6, 945000), u'value': u'http://smile-fnl11.nicovideo.jp/smile?m=29618346.70461'}]}}, u't': 2L, u'v': 2, u'ns': u'nicovideo.posts', u'o2': {u'_id': 29618346}, u'op': u'u'}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 310, in run
    ns, timestamp)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 43, in wrapped
    reraise(new_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 32, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 161, in update
    id=u(document_id))
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 330, in get
    doc_type, id), params=params)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 307, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 93, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 105, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
OperationFailed: TransportError(404, u'{"_index":"nicovideo","_type":"posts","_id":"29618346","found":false}')

看起来像是通过 oplog 找不到对应的数据

在 github 上面提了 issue ，没人回复，所以求助 V 友了

8 条回复 • 2016-09-14 14:48:29 +08:00