当前位置：Gxlcms > 数据库问题 > Logstash7.6.2同步Mysql数据到ElasticSearch

Logstash7.6.2同步Mysql数据到ElasticSearch

时间：2021-07-01 10:21:17 帮助过：7人阅读

&useSSL=false&autoReconnect=true" # 用户名密码 jdbc_user => "user" jdbc_password => "password" # jar包的位置 jdbc_driver_library => "/u02/tomcat/mysql-connector-java-8.0.20/mysql-connector-java-8.0.20.jar" # mysql的Driver jdbc_driver_class => "com.mysql.jdbc.Driver" jdbc_default_timezone => "Asia/Shanghai" jdbc_paging_enabled => "true" jdbc_page_size => "10" #statement_filepath => "config-mysql/test.sql" #注意这个sql不能出现type，这是es的保留字段 statement => "select * from user where name=‘张三‘ " schedule => "* * * * *" } } output { elasticsearch { hosts => "127.0.0.1:9200" # index名 index => "user" # type名 document_type => "_doc" # 需要关联的数据库中有有一个id字段，对应索引的id号 document_id => "%{id}" } stdout { codec => json_lines } }

启动报错记录1： [logstash.config.source.local.configpathloader] No config files found in path {:path=>"/u02/tomcat/logstash-7.6.2/bin/config/sync_tablename.conf"} ./logstash -f config/sync_tablename.conf

报错原因：启动的时候找不到指定路径

解决方法：可以放在bin目录下，启动时不需要指定路径； ./logstash -f sync_tablename.conf

启动报错记录2：[FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", "{", [A-Za-z0-9_-], ‘"‘, "‘", "}" at line 17, column 1 (byte 618) after input {

报错原因：配置文件中17行有错误

解决方法：检查配置文件的内容，是否合规

启动报错记录3：

[ERROR][logstash.inputs.jdbc     ] Invalid setting for jdbc input plugin:

  input {
    jdbc {
      # This setting must be a number
      # Expected number, got "" (type )
      jdbc_page_size => ""
      ...
    }
  }

报错原因：jdbc_page_size必须是一个数字

解决方法：jdbc_page_size => "10"

#验证配置文件命令：
./logstash -f sync_tablename.conf --config.test_and_exit
#显示内容表示配置文件正确
Configuration OK
[INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

4、启动命令进行数据同步 ./logstash -f sync_tablename.conf

启动报错记录1：

Error: unable to load /home/u02/tomcat/logstash-7.6.2/bin/config-mysql/mysql-connector-java-8.0.20/mysql-connector-java-8.0.20.jar from :jdbc_driver_library, file not readable (please check user and group permissions for the path)
  Exception: LogStash::PluginLoadingError

报错原因：插件加载失败，mysql-connector-java插件路径配置错误，

解决方法：去掉/home，改为 /u02/tomcat/logstash-7.6.2/bin/config-mysql/mysql-connector-java-8.0.20/mysql-connector-java-8.0.20.jar

启动报错记录2：

[WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"%{id}", :_index=>"user", :routing=>nil, :_type=>"doc"}, #<LogStash::Event:0x4d04559>], :response=>{"index"=>{"_index"=>"user", "_type"=>"doc", "_id"=>"%{id}", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [user] as the final mapping would have more than 1 type: [_doc, doc]"}}}}
{"@timestamp":"2020-07-02T10:11:23.182Z","message":"","host":"127.0.0.1","@version":"1"}

报错原因：无法将事件索引到Elasticsearch，拒绝对[user]的映射更新，因为最终映射将有一个以上的类型：[_doc，doc]”

解决方法：配置文件中的type要和elasticSearch中的type保持一致

Logstash7.6.2同步Mysql数据到ElasticSearch

标签：文件命令出现文件的错误 ada 地址关联 tps 注意

Logstash7.6.2同步Mysql数据到ElasticSearch

人气教程排行