I was using ELK stack (Elasticsearch, Logstash, Kibana) for FreeIPAPI for a while, but as the traffic and data volume grew, I started facing performance issues and scalability challenges. After researching alternatives, I decided to migrate to ClickHouse for storage and Vector for log processing. In this post, I’ll share my experience and the steps I took to successfully migrate from ELK to ClickHouse and Vector.
For what I was using ELK
FreeIPAPI is receiving more than 100 million requests per day, and I was using ELK stack to store and analyze the logs generated by the API. ELK was a good choice initially because it provided a powerful search and analytics engine with Elasticsearch, a flexible data processing pipeline with Logstash, and a user-friendly visualization tool with Kibana.
It was very important to me to handle the logs via UDP to avoid any performance impact on the API. This means even if the ELK server was down, the API would still be able to receive and process requests without any issues.
Why I decided to migrate
As the traffic and data volume grew, I started facing performance issues with ELK. Elasticsearch was struggling to keep up with the write load, and the search performance was degrading. Additionally, managing and scaling ELK stack was becoming increasingly complex and costly.
And of course, I needed to upgrade my setup because I was using the older versions of ELK as it was FREE.
What was my server setup
For ELK I was using a single server with 8GB RAM and 4 CPU cores. It was running Elasticsearch, Logstash, and Kibana all together. This setup was sufficient for a while, but as the traffic increased, it became clear that I needed a more scalable solution.
Keeping in mind that the CPU was 100% utilized all the time!
Why ClickHouse and Vector
After researching alternatives, I decided to migrate to ClickHouse for storage and Vector for log processing. ClickHouse is a columnar database that is optimized for analytical queries and can handle large volumes of data with high performance. Vector is a high-performance log processing tool that can efficiently collect, transform, and route logs to various destinations.
Also, Vector supports UDP input, which was crucial for my use case to ensure that the API performance is not impacted by log processing.
Migration Steps
I changed nothing on my application servers! this was very important part.
Then I took down the ELK setup! and as I said, UDP logging helped me to keep the API running without any issues.
Then I booted up my new docker-compose setup with ClickHouse and Vector on the same server, with same UDP port for logging.
And booom! everything started working immediately! I was able to see the logs being processed by Vector and stored in ClickHouse without any issues.
Performance Improvements
So far that I am writing to you, The server is only using 4% of its CPU and 1GB of RAM! which is a huge improvement compared to the ELK setup where the CPU was at 100% all the time.