Logpresso Enterprise - Real-time processing and external integration

Real-time data processing, solved with Logpresso!
Experience powerful big data solutions.

Introduction to the course

How to process real-time data?

Today's new data processing trends, such as Data Lake, are based on real-time data processing methods based on big data. However, building a real-time big data system is very difficult. Not only is it expensive, but even if you try to configure it based on open source or cloud, it involves the work of integrating multiple systems rather than a single solution.

Logpresso is a platform designed to be highly suitable for real-time data processing. By installing Logpresso on a single device, you can perform all aspects of real-time processing, including collection, preprocessing, and storage.

This lecture!

This course will teach you how to perform data engineering, including real-time processing and external integration, which are the special features of Logpresso. By deeply explaining everything that Logpresso can do in real time, the goal is to help you build a real-time big data processing platform with relatively little time and effort.

Data such as logs must contain the truth of the moment to be useful. To do this, you need to know the tools and methods for referencing real-time data. In this lecture, we will deeply discuss CEP (Complex Event Processing, real-time event context) to detect various actions based on the chronological relationship and collection interval of logically related cases in real time or to perform follow-up work. In addition, we will teach you how to easily perform external linkage by processing the collected big data and sending it to files, DBs, networks, etc.

Through this lecture, you will develop a perspective that allows you to view the overall real-time processing function from a data engineering perspective .

What to prepare

PC or laptop

• You will need a Windows-based PC or laptop.

Chrome based browser

• Logpresso allows you to perform all queries through the web console.
• It can be used without any problems in browsers that support HTML5, such as IE 11 or higher, Chrome series (Edge, Whale, etc.), Firefox, Safari, etc. (However, Chrome series is recommended.)

SFTP Server

• In Section 2 [External Data Linking], you will practice configuring a local SFTP server for remote file access.
• For this purpose, we either utilize accessible servers or use freeware provided by links.

Database

• In Section 2 [External Data Linking], you will practice querying to import or export data from a database. For this, you will need a database.
• As long as you have a database that can read and write data, you will have no problem progressing through the lecture.
• This lecture will proceed using MariaDB, an open source database that is already installed. Therefore, a separate installation process will not be covered.

Tools covered in the lecture

• This course uses the Logpresso Enterprise Server version.
• We also use the data provided for lectures in Logpresso to conduct lectures. Tools and data can be accessed through the lectures provided in Section 0 [Preparing for the Lecture] after registering for the lecture.
• You may need to install a separate DB for database connection.

Expected Questions Q&A 💬

Associated Processes

Getting started with Logpresso
First time using Logpresso
What if you're an engineer?

Logpresso Enterprise -
Data collection and parsing
Data collection and normalization

Logpresso Enterprise -
Batch processing
All about the reservation query feature