Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [debug] [out coro] cb_destroy coro_id=9 [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/metrics-server-7b4f8b595-v67pp_kube-system_metrics-server-e1e425c84b9462fb800c3655c86c1fd8320b98067c0f43305806cb81b7120b4c.log, inode 67182317 Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [outputes.0] HTTP Status=200 URI=/_bulk [2022/03/24 04:20:49] [debug] [outputes.0] task_id=0 assigned to thread #1 (fbit, es) Make sure you're using either 1.9.1 or 1.8.15. * [2022/03/24 04:19:29] [debug] [http_client] not using http_proxy for header [2022/03/24 04:19:34] [debug] [outputes.0] task_id=1 assigned to thread #0 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"0OMmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"2eMmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. is any way to skip create index if exists? Share. Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [ warn] [engine] failed to flush chunk '1-1648192113.5409018.flb', retry in 8 seconds: task_id=11, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [upstream] KA connection #36 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [http_client] not using http_proxy for header [2022/03/24 04:20:20] [debug] [input:tail:tail.0] inode=69179340 removing file name /var/log/containers/hello-world-6lqzf_argo_wait-6939f915dcb1d1e0050739f656afcd8636884b83c4d26692024699930b263fad.log "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"5-Mmun8BI6SaBP9luq99","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall [2022/03/24 04:21:20] [debug] [input:tail:tail.0] inode=1885001 removing file name /var/log/containers/hello-world-g74nr_argo_wait-227a0fdb4663e03fecebe61f7b6bfb6fdd2867292cacfe692dc15d50a73f29ff.log, tried 1.9.0, 1.8.15. After Graylog was created. . Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [retry] re-using retry for task_id=6 attempts=2 [2022/03/24 04:19:22] [ warn] [engine] failed to flush chunk '1-1648095560.297175793.flb', retry in 7 seconds: task_id=2, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/traefik-5dd496474-84cj4_kube-system_traefik-686ff216b0c3b70ad7c33ceddf441433ae1fbf9e01b3c57c59bab53e69304722.log, inode 34105409 [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/svclb-traefik-twmt7_kube-system_lb-port-443-ab3854479885ed2d0db7202276fdb1d2142db002b93c0c88d3d9383fc2d8068b.log, inode 34105877 Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [retry] new retry created for task_id=14 attempts=1 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Setting Type doc in the es OUTPUT helped in my case. Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [retry] re-using retry for task_id=3 attempts=2 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"4-Mmun8BI6SaBP9luq99","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:19:24] [ warn] [engine] failed to flush chunk '1-1648095560.254537600.flb', retry in 10 seconds: task_id=1, input=tail.0 > output=es.0 (out_id=0) "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"deMnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [OUTPUT] Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [upstream] KA connection #35 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) [2022/03/24 04:19:34] [debug] [upstream] KA connection #104 to 10.3.4.84:9200 has been assigned (recycled) Logstash_Format On Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [ warn] [engine] failed to flush chunk '1-1648192100.653122953.flb', retry in 17 seconds: task_id=3, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:19:38] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"5uMmun8BI6SaBP9luq99","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Hi! Fri, Mar 25 2022 3:08:47 pm | [2022/03/25 07:08:47] [debug] [out coro] cb_destroy coro_id=17 Name es Improve this answer. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"3-Mmun8BI6SaBP9luq99","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [retry] re-using retry for task_id=9 attempts=2 [2022/04/17 14:48:10] [ warn] [engine] failed to flush chunk '1-1650206880.316011791.flb', retry in 16 seconds: task_id=4, input=tail.0 > output=es.0 (out_id=0 . Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [ warn] [engine] failed to flush chunk '1-1648192108.829100670.flb', retry in 8 seconds: task_id=7, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=3070975 file has been deleted: /var/log/containers/hello-world-hxn5d_argo_wait-be32f13608de76af5bd4616dc826eebc306fb25eeb340049de8d3b8e5d40ba4b.log "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"1OMnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=3076476 events: IN_ATTRIB Match host. [2022/03/22 03:57:48] [ warn] [engine] failed to flush chunk '1-1647920205.172447077.flb', retry in 912 seconds: task_id=12, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [task] created task=0x7ff2f183b560 id=20 OK Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [retry] re-using retry for task_id=16 attempts=2 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/coredns-66c464876b-4g64d_kube-system_coredns-3081b7d8e172858ec380f707cf6195c93c8b90b797b6475fe3ab21820386fc0d.log, inode 67178299 [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=34055641 removing file name /var/log/containers/hello-world-bjfnf_argo_main-0b26876c79c5790bdaf62ba2d9512269459746b1c5711a6445256dc5a4930b65.log [2022/03/24 04:19:34] [debug] [http_client] not using http_proxy for header Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:47 pm | [2022/03/25 07:08:47] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [task] created task=0x7ff2f183ade0 id=16 OK [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Enabling debug logging in fluentbit should give more info. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"PeMmun8BI6SaBP9luqVq","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall I only changed the output config since its a subchart. [2022/03/24 04:19:59] [debug] [upstream] KA connection #102 to 10.3.4.84:9200 has been assigned (recycled) [2022/03/24 04:19:49] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [retry] re-using retry for task_id=0 attempts=3 Btw, although some warn messages, I still can search specific app logs from elastic search. It's possible for the HTTP status to be zero because it's unparseable -- specifically, the source uses atoi () -- but flb_http_do () will still return successfully. retry_time=29 next_retry_seconds=2021-04-26 15:58:43 +0000 chunk . Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:22] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=104048677 removing file name /var/log/containers/hello-world-hxn5d_argo_main-ce2dea5b2661227ee3931c554317a97e7b958b46d79031f1c48b840cd10b3d78.log [2022/03/24 04:19:24] [debug] [out coro] cb_destroy coro_id=0 What version? "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"l-Mmun8BI6SaBP9l7LpS","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ffffhello-world-dcqbx_argo_wait-6b82c7411c8433b5e5f14c56f4b810dc3e25a2e7cfb9e9b107b9b1d50658f5e2.log, inode 67891711 Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=1931990 watch_fd=19 Fri, Mar 25 2022 3:08:37 pm | [2022/03/25 07:08:37] [debug] [retry] re-using retry for task_id=7 attempts=2 Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [http_client] not using http_proxy for header "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"f-Mnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall I have deployed the official helm chart version 0.19.23. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"eOMnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:21:20] [debug] [input:tail:tail.0] inode=35326802 with offset=0 appended as /var/log/containers/hello-world-wpr5j_argo_main-55a61ed18250cc1e46ac98d918072e16dab1c6a73f7f9cf0a5dd096959cf6964.log I'm using M6g.2xlarge (8 core and 32 RAM) AWS instances 3 master and 20 data nodes. Logstash_Format On Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [ warn] [engine] failed to flush chunk '1-1648192121.87279162.flb', retry in 10 seconds: task_id=15, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [retry] re-using retry for task_id=4 attempts=2 [2022/03/24 04:20:26] [debug] [upstream] KA connection #102 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [outputes.0] HTTP Status=200 URI=/_bulk "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"OOMmun8BI6SaBP9luqVq","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [ warn] [engine] failed to flush chunk '1-1648192110.850147571.flb', retry in 11 seconds: task_id=9, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:46 pm | [2022/03/25 07:08:46] [debug] [outputes.0] task_id=12 assigned to thread #1 Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [outputes.0] task_id=9 assigned to thread #1 [2022/03/24 04:20:36] [debug] [out coro] cb_destroy coro_id=6 Feel free to reopen if you think this is something we should be able to fix on the operator side. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [outputes.0] task_id=21 assigned to thread #0 Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [ warn] [engine] failed to flush chunk '1-1648192109.839317289.flb', retry in 8 seconds: task_id=8, input=tail.0 > output=es.0 (out_id=0)