Fluentd is a cross-platform open-source data collection software project originally developed at Treasure Data. It is written primarily in the C programming language with a thin-Ruby wrapper that gives users flexibility.[2]
Overview
Fluentd was positioned for "big data," semi- or un-structured data sets. It analyzes event logs, application logs, and clickstreams.[3] According to Suonsyrjä and Mikkonen, the "core idea of Fluentd is to be the unifying layer between different types of log inputs and outputs.",[4] Fluentd is available on Linux, macOS, and Windows.[5]
History
Fluentd is written by Sadayuki Furuhashi under the development of the Treasure Data company. It was designed in 2011 and published as open source software in October 2011.[6] In 2013, the company announced the funding of $5 million. Treasure Data was acquired by Arm Limited (now known as ARM Limited). Arm] in 2018.[7] Fluentd was accepted as a hosted project by the Cloud Native Computing Foundation (CNCF) in November 2016.[8]Fluentd has passed from the CNCF's incubation program on 11 April 2019.[9] It has been continued developing after graduation, and Fluentd releases in the 2020s. In 2024, Fluentd versions 1.16.4, 1.16.5, 1.17.0, 1.17.1 and 1.18.0 were released, demonstrating continued active development.[10]
Users
Fluentd was one of the data collection tools recommended by Amazon Web Services in 2013, when it was said to be similar to Apache Flume or Scribe.[11] Google Cloud Platform's BigQuery recommends Fluentd as the default real-time data-ingestion tool, and uses Google's customized version of Fluentd, called google-fluentd, as a default logging agent.[12][13]
Fluent Bit
Fluent Bit is a log processor and log forwarder that is being developed as a CNCF sub-project under the umbrella of Fluent project.[14] Fluentd is written in C and Ruby and consumes at least 60 megabytes of memory. Fluent Bit is written only in C, with no dependencies, and consumes approximately one megabyte of memory, making it easier to run under embedded Linux and in containers.[15]
References
- ^ "Releases - fluent/fluentd" – via GitHub.
- ^ "Overview | Fluentd".
- ^ Pasupuleti, Pradeep and Purra, Beulah Salome (2015). Data Lake Development with Big Data. pp. 44–45; 48. Packt. ISBN 1785881663
- ^ Suonsyrjä, Sampo and Mikkonen, Tommi "Designing an Unobtrusive Analytics Framework for Monitoring Java Applications", pp. 170–173 in Software Measurement. Springer. ISBN 3319242857
- ^ Fluentd.org. "Download Fluentd". Retrieved 10 March 2016.
- ^ "What is Fluentd?". Fluentd. Retrieved 30 September 2026.
- ^ "Arm Acquires Treasure Data to Set the Stage for IoT Transformation". Arm. 2 August 2018. Retrieved 30 September 2026.
- ^ "Fluentd joins the Cloud Native Computing Foundation". Cloud Native Computing Foundation. 9 November 2016. Retrieved 30 September 2026.
- ^ "Cloud Native Computing Foundation announces Fluentd graduation". Cloud Native Computing Foundation. 11 April 2019. Retrieved 30 September 2026.
- ^ "Fluentd release archive". Fluentd. Retrieved 30 September 2026.
- ^ Parviz Deyhim (August 2013). "Best Practices for Amazon EMR" (PDF). Amazon Web Services. p. 12. Archived from the original (PDF) on 2016-03-26. Retrieved March 24, 2017.
- ^ Google Cloud Platform (2016). "Real-time logs analysis using Fluentd and BigQuery". Retrieved 10 March 2016.
- ^ Google Cloud Platform (2016). "The Logging Agent". Retrieved 10 March 2016.
- ^ "Fluent Bit". fluentbit.io. Retrieved 2021-12-05.
- ^ "Fluentd & Fluent Bit - Fluent Bit: Official Manual". Retrieved 2021-12-05.
Further reading
- Goasguen, Sébastien (2014). 60 Recipes for Apache CloudStack: Using the CloudStack Ecosystem, "Chapter 6: Advanced Recipes". O'Reilly Media. ISBN 1491910127
- Wilkins, Phil (2022). Logging in Action, With Fluentd, Kubernetes and more. Manning. ISBN 9781617298356
