Log Management Installation & Configuration

Log Management - Log Forwarder and Collector Installation

Ceburu’s Log Management module supports a distributed architecture using Log Forwarders and Log Collectors for efficient log ingestion and routing. 

Log Forwarder and Collector Installation Requirements:

The following prerequisites must be met to install and operate the Log Forwarder and Collector successfully.

Click the below link to see the System requirements:

CeburuAI Installation and Configuration

Architectural Diagram:

Forwarders handle log transport between agents and collectors, while collectors perform aggregation and normalization before sending logs to the analytics engine. 

1. Access the Log Management Configuration

• Navigate to Configurations → Log Management.

• You will see two sections:

  • Log Forwarders Configuration

  • Log Collectors Configuration

2. Setup a Log Forwarder

1. Click the Setup Log Forwarder button (top-right corner).

2. In the setup window:

   • Select Device: Choose the device where the forwarder will be installed (for example, your Primary Probe).

   • Forwarder Port: Specify the port number (default is 24224).

     Note: The port cannot be changed after the forwarder is created.

3. Click Install Forwarder to begin deployment.

Once the installation is complete:

• The forwarder will appear under Log Forwarder Configuration.

• Its status will update to Installed and then Active once connected successfully.

• Install multiple forwarders for redundancy or for different network zones.

Install Log Collectors

After setting up the forwarders, you can deploy Log Collectors to capture and process logs.

There are two installation options:

Option A: Install Individually

1. Under the Log Collectors section, locate the device where you want to install the collector.

2. Click the Download icon next to the device entry.

3. The collector will automatically install and begin connecting to the assigned forwarder.

Option B: Bulk Installation

1. Select multiple devices in the Log Collectors section.

2. From the top-right dropdown menu, choose Deploy Collector.

3. The collectors will be installed on all selected systems simultaneously.

Verify Deployment:

Once installation is complete:

• Forwarders will appear under Log Forwarder Configuration with a status of Active.

• Collectors will appear under Log Collectors with the Collector Status showing Active.

• If any collector or forwarder shows as Inactive or Failed, click the restart icon under the Actions column to retry or troubleshoot.

Configure Log Source:

Go to the Log Management tab. Click on Add Log Source in the Log Sources section.

Log sources define where logs are collected from and how they are ingested into the system. Each log source configuration specifies the source type, collection parameters, and the target machines where the log collector is deployed. 

Enter Log Source Details

Identifier Tag

The identifier tag is used to index and query logs across the log management system.

Rules:

• No spaces allowed

• Allowed characters include uppercase and lowercase letters, numbers, and special characters
  - _ ~ ! @ # $ & * + ; : , . ? < > \ | /

Usage:

• Used for filtering, searching, and identifying log data across the platform

Examples:

• production-webapp

• staging_api

• tag1-23

• Production@2024

• API_Service#1

Querying Tags

Querying tags are key value pairs attached to all logs collected from a source. They help filter and search logs across services and machines.

Format:

• Stored as JSON key value pairs
  Example: "environment": "prod"

Character Rules:

• Only lowercase letters, numbers, hyphens, and underscores are allowed

Multiple Tags:

• Multiple tags can be added to categorize logs

Editing and Deletion:

• Tags cannot be edited after creation

• To change a tag, delete and re add it

• Deleting a tag only affects future logs, not existing data

Examples:

• "environment": "production"

• "service": "webapp"

• "category": "application"

Specify Log File Configuration

Files and Folders:

Used to collect logs directly from files on a system.

Key Settings:

• File Path: Absolute path to the log file
  Example: /var/log/application.log

• File Pattern: Wildcards to match multiple files
  Example: /var/log/*.log

• Encoding: File encoding format, default UTF 8

• Parser: Select based on log format

• Multiline Pattern: Optional regex for multiline logs

• Advanced Parameters: Tune performance and memory usage

Syslog:

Used to receive logs over the network from servers and network devices.

Key Settings:

• Mode: UDP or TCP transport protocol

• Port: Listening port for syslog messages
  Default commonly used port is 514

• Parser: RFC 5424 for modern syslog or RFC 3164 for legacy formats

• Advanced Parameters: Optimize handling of high volume log streams

HTTP Events:

Used when applications push logs using HTTP POST requests.

Key Settings:

• Listen Address: Fixed to 0.0.0.0 to accept requests from all interfaces

• Port: Port for receiving HTTP log events

• Parser: JSON parser for structured payloads

• Advanced Parameters: Configure buffer sizes and response handling

• Successful Response Code: Default is 201

Deployment:

Select the machines where the log collector will be installed.

• Multiple machines can be selected

• Deployment status is tracked per machine

• Failed deployments can be retried
  

Once added, the new log source will appear in the Log Sources list for monitoring and analysis.

Make sure the file path and permissions are correct on the target device to avoid collection errors.