Starts on 7 Agustus 2024

Rabu dan Ahad

20.00-22.00

Syllabus:

Meeting 1: Introduction to Data Engineering

- Overview of Data Engineering: Roles, responsibilities, and importance.

- Data Lifecycle: Ingestion, storage, processing, and visualization.

- Tools and Technologies: Introduction to popular tools (e.g., SQL, NoSQL, ETL tools, Hadoop, Spark, Kafka).

Meeting 2: Data Ingestion and Storage

- Data Ingestion Techniques: Batch vs. real-time ingestion.

- Introduction to Databases: SQL vs. NoSQL, key concepts, and differences.

- Hands-on Activity: Setting up a simple database and ingesting data.

Meeting 3: Data Processing with SQL

- Advanced SQL Queries: Joins, aggregations, subqueries.

- Data Transformation: Techniques for transforming raw data into usable formats.

- Hands-on Activity: Writing and executing complex SQL queries.

Meeting 4: Big Data Technologies

- Introduction to Hadoop: HDFS, MapReduce.

- Introduction to Spark: Core concepts, RDDs, DataFrames.

- Hands-on Activity: Running simple Spark jobs on a local cluster.

Meeting 5: Data Pipelines and ETL Processes

- ETL Concepts: Extract, Transform, Load processes.

- Building Data Pipelines: Tools (e.g., Apache Airflow, Luigi).

- Hands-on Activity: Creating a basic ETL pipeline using a selected tool.

Meeting 6: Real-time Data Processing

- Introduction to Kafka: Core concepts, producers, consumers.

- Streaming Data Processing: Using tools like Apache Flink or Spark Streaming.

- Hands-on Activity: Setting up a Kafka producer and consumer, processing real-time data.

Meeting 7: Data Visualization and Reporting

- Data Visualization Tools: Tableau, Power BI, or open-source tools.

- Creating Dashboards: Best practices for effective data visualization.

- Hands-on Activity: Building a simple dashboard using a selected tool.