StatsD vs CollectD: Most detailed version

Busy servers and applications have a lot of things to monitor and there are stats in several forms. The stats are regarding your servers, applications running on those servers and loads of metrics which needs to be collected and monitored properly. Processing and collecting these stats help in deciding factors like scaling, performance of the system, troubleshooting and among more in your configuration. For monitoring to be precise, the system needs loads of data and the amount of data collected will likely increase the chances of understanding what is happening at any point of time. In this blog, we will be placing two popular collection programs StatsD and CollectD head to head to see what works in diverse use cases, and listing their pros and cons. Let’s go over why StatsD and CollectD are called daemons.

Daemons-:

A daemon is a sort of program running continuously and it handles the periodic service requests that a computer system expects to receive. All the requests are then transferred to all the appropriate programs by these daemons.

Out of several examples, they have HTTPD(Hypertext Transfer Protocol Daemon) for web pages that stays awake waiting for collecting all the requests from Web clients and their users. It also collects data from your system. These types of processes usually ends with “D” in the end to differentiate them with other processes running in the background.

Now working of these daemons might sound very simple but it is a part of more complex architecture based on these 4 key points:

Source – Where is the information is coming from?
Transport – How will this information reach you?
Storage – Where will this data be stored?
View – How will you see what’s going on in your system?

Meet StatsD:

StatsD is a type of daemon, designed by engineers at Etsy. The primary task achieved using StatsD is to aggregate and summarize application metrics. It was developed to help the team transmit data points at their networks and it helped them turning the information into graphs. The usage of StatsD is growing very fast. While operating StatsD always looks for simple UDP packets( also represents a uni-data point) through an interface. This signifies that it can allow a huge amount of data and information in a connectionless way. Then it can integrate the values and forward them to Graphite. In simple words, all we have to do is show what initial metrics we want to create by sending all the related data and wait for the information to flow through UDP port.

Architecture view:

Pros:

It is very simple and efficient to use. It handles aggregation as well. The StatsD protocol is text-based and straightforward to write and read.
It is an Open Source project and it is easily available for use at Github.
StatsD allows you to send in a large amount of stats without worrying about increasing application latency.
There are many client-side libraries in several programming languages by which you can send data directly to StatsD instance. You have StatsD clients for Java, Erlang, Python, Ruby, Node, Go, Haskell or broadly every other language.
StatsD clients are very thin and light, it carries no state, need no threads

Cons:

When using StatsD server you need to input the minimum network configuration required. And it precisely means that you have to deal with the volatile IPs, network segmentation and routing etc.
Initially, in the metrics, all the information about the entity(host, pod, container, process) is lost unless you format the metrics to explicitly hold that data. These kinds of problems arise when you want to deploy containers and microservices, where all the entities regarding the software are volatile and you are bound to group them as a functionality.
The most arising problem with StatsD server is that it acts as a single point failure. If it crashes or and suffers from any problem then all the monitoring and visualization process is temporarily down. The UDP packages for those specific periods are also lost.

Meet CollectD:

CollectD is a type of daemon, actively used for collecting detailed information about a server. It is one of the easiest ways to gather statistics about several different components of a server environment. Tracking common metrics like usage of memory, network traffic, CPU load etc. has become highly efficient and easy with the help of CollectD. Using these common metrics you can easily correlate events with the state of your system.

CollectD is developed and written in C, which allows it to run on a system. In CollectD plugins play a very important role as all the task is done by using them. Currently, there are over 90 plugins that range from standard case to low level, specialized and advanced topics. By using these plugins you can easily track common software like Apache, Nginx, iptables, Memcache, PostgreSQL and many more.

Architecture View:

Pros:

Portability and performance are one of the key features CollectD was developed to. The daemon stays in memory, so you don’t need to restart a heavy interpreter every time new values are logged in the system.
CollectD has a multi-threaded layout which allows the daemon to have multiple plugins up and running simultaneously without causing any error due to any Input-output latencies. It is built to scale and it can handle any number of hosts.
The designing of the protocols is very flexible and future proof. It is designed lightweight so the data collection over slow connections will not create any kind of problems. The extensibility of the protocol opens it for new features in the future without hindering the backward compatibility.
Advance technologies like IPv6 and Multicast are available to use by the network code.

Cons:

The usage of plugins at runtime is extensible, The daemon needs to restart every time to use other new plugins.
CollectD is more of a plugins dependent tool, depending on plugins for every beginner and advanced topic. If the plugins are not present, using it will be quite tough.
It has no support for Cisco’s IP Service level Agreement mechanism. There is no standard way defined to get trending network data over time. And its algorithm is not designed in a way to predict trend over network data.
It has no Logical Grouping. It means, no support arranging the host or devices it monitors into user-defined groups.

Comparing StasD and CollectD:

Nature: StatsD is more of a library which aggregate metrics and relay them to virtually any graphing or monitoring backend. Implementing StatsD is very simple and efficient. StatsD is used more for application monitoring. You can send custom metrics in a set interval of time. StatsD is perfect for this because it is a general-purpose daemon that accepts UDP traffic.On the other hand, CollectD collects, transfers and stores system performance statistics, with data acquisition and storage handled by plugins. Plugins are the rockstar of this daemon, they handle almost every type of task for CollectD. And of course, a StatsD plugin. You can connect the two plugins and use them to report different events. Now, this is a plus point as CollectD is not a monitoring technology, all you can do with it right now is threshold checking.

Architecture: When it comes to the architecture both the daemons have different approaches. Architecture is a collection of functionalities in order. StatsD has a very simple and linear structure. In StatsD the story begins with the initial metrics and ends with the resulting visualizations using Graphite and other visualization tools. The architecture is light in weight so there is no need for threads and states.

StatsD has a very simple and linear structure. In StatsD the story begins with the initial metrics and ends with the resulting visualizations using Graphite and other visualization tools. The architecture is light in weight so there is no need for threads and states. On the other hand, CollectD has few more layers introduced in its architecture as we call them plugins. These plugins widen the usage of the collectD daemon. All the plugins are connected to collectD and then RRDtool, CSV plugins for storing data and statistics. There are over 90 plugins currently for a variety of tasks and it also has a multithreaded layout which means various plugins can be introduced at once and they can run simultaneously without any I/O latencies.

Availability: Both the daemons are easily available and are Open-source. Anyone can start using them by visiting Github. CollectD has 2.4K GitHub stars and 1.1K GitHub forks and you can visit the repository from here. StatsD is also quite popular on GitHub. You can check out the repository from here.

Graphite-:

Graphite is a graphing library which comprises of several other parts that are integrated to help in rendering and updating all the visual depictions of the data concerning time. It runs on both cheap hardware or the Cloud infrastructure. Usage of graphite is very popular between teams as a monitoring device for there websites and applications.

For setting up and using Graphite on Ubuntu you can refer to this blog.

Graphite has already been briefed as a graphing library solely responsible for storing and rendering a visual representation of data. This means that for all the other operations on metrics like collect and transfer the data points, Graphite requires the help of other applications. Graphite project itself has several components, each of them has specific and definite goals.

Graphite web app – is the visible and dynamic components of the Graphite installation. It is where the representation and graph design takes place. Often termed as the face of Graphite.
Carbon – plays as the storage backend for Graphite configuration. It can accept time series data over a common set of protocols. Often, known as Carbon daemon.
Whisper – is the database library used by Graphite to store all the data that is sent to it.

How StatsD and CollectD are used with Graphite:

Graphite only works towards rendering and plotting visual representation of metrics. It cannot collect data per se. Hence, that’s where the use of daemons arises. In this case, StatsD and CollectD can act as the perfect collection programs for Graphite.

In StatsD, The collection of initial metrics is first transferred through a UDP port in the form of packets, that are represented as single data points. It means a lot of data and information is welcome in a connectionless way. Then StatsD performs its role of aggregating and integrating and then it transfers it to Graphite for further representations.

In CollectD, The approach is almost the same as StatsD i.e. flushing the data to Graphite. But after setting up and configuring CollectD we need to work on the plugins part. The plugins are directly connected to the CollectD program and now it’s the time to choose the services CollectD will gather information about using these plugins. After collection then the data is again sent for further representation and operations on it.

CONCLUSION -:

In conclusion, the two daemons have different ways of proving their worth. Each of them having their respective pros and cons helping you in differentiating and making the right decision while using them for your own use case. An all-round understanding of how collection of metrics work, and how Graphite visualises those metrics also helps in making the right choice.

REFERENCES-:

ABOUT THE AUTHOR:-

Kaushlendra Pratap is a Flutter and Python Developer. He loves contributing to open-source and he is an active contributor since 2017 both online & offline. Writing technical content is his hobby and he is a full-time writer at Mixster. He writes prose in noon and code at night. You can find me on LinkedIn and Twitter.

Leave a Reply Cancel reply