Elasticsearch 기반 대규모 로그 데이터의 인덱싱 및 검색 성능 최적화

초록

In the era of big data, efficiently storing and analyzing large-scale log data in real time is crucial, and Elasticsearch performance depends heavily on configuration settings. This study empirically optimizes Elasticsearch and Logstash configurations to maximize indexing and search efficiency. Results show that indexing performance is driven by CPU utilization and data processing strategies such as batch size, heap size, and node distribution. Rather than merely increasing resources, effective workload balancing and resource allocation are essential. Optimizing Logstash parallel processing and distributing memory and CPU resources strategically are key, while excessive shard allocation degrades performance. Search performance is significantly affected by cache utilization, sorting mechanisms, and data types, with unnecessary sorting being a major bottleneck. Minimizing these operations greatly improves query speed. Additionally, performing indexing and search on the same index causes resource contention, whereas separating them into distinct indices mitigates interference and enhances stability. While index sorting improves specific search scenarios, it introduces indexing overhead, requiring careful application. Applying the optimized configurations led to a 9.8x increase in indexing speed and a 10x improvement in search performance.

키워드