CloudLab: Hybrid Cloud ELK Stack for Home Network Monitoring

Introduction

In an era where digital security is paramount, CloudLab emerged as a beacon of innovation. This project leverages a hybrid cloud solution with a containerized ELK stack deployed on a Google Cloud VM to monitor my home network. CloudLab is not just a technical achievement; it’s a testament to the power of modern, scalable monitoring solutions.

The Concept and Setup

The concept was straightforward—deploy a robust monitoring solution using the ELK stack in a hybrid cloud environment. Utilizing a Google Cloud VM for the ELK stack ensures scalability, flexibility, and efficient resource utilization. Suricata, installed locally on ShadowDragon, sends logs to the ELK stack via Filebeat, while Logstash collects syslogs from the home router, seamlessly integrating all components.

Technical Details

Google Cloud Compute Engine:
Central hub for the ELK stack deployment.
Elasticsearch:
Centralized logging and search engine.
Logstash:
Data processing and transformation.
Kibana:
Visualization and dashboard creation.
Suricata:
Network threat detection engine, installed locally on ShadowDragon, with logs sent via Filebeat.
Filebeat:
Acts as a go-between for Suricata and the ELK stack.

System Specs

Infrastructure:

• Platform: Google Cloud VM (CloudLab instance)
• Containerization: Docker
• Orchestration: Docker Compose

System Specs:

Machine type: e2-standard-2
CPU platform: Intel Broadwell
Architecture: x86/64

Installation Playbook

Environment Setup:

Provisioning the Google Cloud VM for CloudLab.
Setting up Docker and Docker Compose on the VM.

ELK Stack Deployment:

Configuring and deploying Elasticsearch, Logstash, and Kibana containers on the VM.
Setting up Suricata locally on ShadowDragon and using Filebeat to forward logs to the cloud.
Logstash configuration on the VM to collect syslogs from the home router.

Integration and Configuration:

Connecting Suricata to Logstash via Filebeat.
Configuring Logstash on the VM to parse and send data to Elasticsearch.
Designing Kibana dashboards for real-time monitoring.

Challenges and Solutions

1. Configuration Issues:
Initial misconfigurations in the ELK stack.

Developed a detailed approach to resolve each configuration issue on the VM and local network. This involved systematic testing and verification of each component in the ELK stack to identify and rectify misconfigurations.

Certificate Path Problems:
Encountered during secure communications setup on the VM.

Implemented a robust certificate management strategy, ensuring secure communication between components of the ELK stack. This included generating, distributing, and renewing SSL certificates as needed.

Network Connectivity:
Struggled to connect CloudLab VM to the home network, initially relying on Tailscale before resolving with careful firewall, port configurations, and static IP addresses.

Eliminated the need for third-party software by meticulously configuring firewalls, ports, and static IP addresses. This ensured seamless hybrid cloud connectivity by setting up appropriate firewall rules, port forwarding, and assigning static IPs to critical components.

Log Ingestion Delays:
Optimizing Logstash for better performance.

Adjusted Logstash settings to ensure timely log ingestion. This involved optimizing pipeline configurations, increasing JVM heap size, and fine-tuning filter plugins to handle high-volume data efficiently.

1. Configuration Issues:
Initial misconfigurations in the ELK stack.

Developed a detailed approach to resolve each configuration issue on the VM and local network. This involved systematic testing and verification of each component in the ELK stack to identify and rectify misconfigurations.

Certificate Path Problems:
Encountered during secure communications setup on the VM.

Implemented a robust certificate management strategy, ensuring secure communication between components of the ELK stack. This included generating, distributing, and renewing SSL certificates as needed.

Network Connectivity:
Struggled to connect CloudLab VM to the home network, initially relying on Tailscale before resolving with careful firewall, port configurations, and static IP addresses.

Eliminated the need for third-party software by meticulously configuring firewalls, ports, and static IP addresses. This ensured seamless hybrid cloud connectivity by setting up appropriate firewall rules, port forwarding, and assigning static IPs to critical components.

Log Ingestion Delays:
Optimizing Logstash for better performance.

Adjusted Logstash settings to ensure timely log ingestion. This involved optimizing pipeline configurations, increasing JVM heap size, and fine-tuning filter plugins to handle high-volume data efficiently.

Technical Arsenal

Tools:

• Elasticsearch: For indexing and searching logs, deployed on Google Cloud VM.
• Logstash: For processing and transforming log data, deployed on Google Cloud VM.
• Kibana: For visualizing data and creating dashboards, deployed on Google Cloud VM.
• Suricata: For detecting network threats and generating logs, installed locally on ShadowDragon.
• Filebeat: For forwarding Suricata logs to Logstash on the cloud VM.
• Docker: For containerizing the ELK stack components on the VM.
• Tailscale: Initially used for network connection troubleshooting before final network configuration.

Impact and Applications

Use cases:

• Real-Time Monitoring: Providing immediate insights into network activities and potential threats through a hybrid cloud setup.

• Home Network Security: Enhancing the security and visibility of home network operations using a scalable cloud solution.

Future Project:

Exploring the potential of Elasticsearch for monitoring data-heavy environments and using algorithms to find relations between different data sources.

CloudLab exemplifies how modern monitoring solutions can be effectively deployed and managed using hybrid cloud environments. This project not only highlights technical proficiency but also underscores the importance of robust network security.