Multimodal Data Extraction ETL Pipeline for Fake News Detection

Python, Data Engineering, ETL, Apache Airflow, Web Scraping, Streamlit, NLP, Computer Vision, Data Pipeline

Multimodal Data Extraction Pipeline for Fake News Detection

Overview

This project implements the design and industrialization of a multimodal data pipeline (text + image), from source identification to production monitoring. The objective is to feed the analysis engine of a disinformation-fighting startup with structured news publications, systematically combining text and image, extracted in an automated and reproducible manner.

The project applies Data Engineering best practices to ensure the reliability, traceability, and scalability of an ETL pipeline designed to train AI models.

Objectives

• Identify and qualify multiple multimodal data sources relevant to disinformation.
• Develop automated extraction scripts that are modular and executable without manual intervention.
• Build a reproducible and logged transformation pipeline, documented by a conceptual schema.
• Orchestrate the entire flow via Apache Airflow (automated ETL).
• Define quality and performance KPIs, and a monitoring plan for production use.

Skills Demonstrated

Data Sourcing and Qualification

• Identification of multiple multimodal sources (APIs, social networks, open data bases, RSS feeds) with evaluation of format, language, label quality, and usage rights.
• Rigorous distinction between controversial opinion (subjective) and disinformation (objectively false), critical for the reliability of training data.
• Prioritization of official access channels (API) before resorting to scraping, with systematic verification of legality.
• Production of a source exploration report formalizing the chosen extraction methods.

Automated Data Extraction

• Development of modular Python scripts (connection, parsing, cleaning, saving) for the extraction of associated text and images.
• Management of API quotas, errors (try/except), and configuration parameters.
• Validation of the correct association between each text and its image, an essential condition for multimodal exploitation.
• Logging of each script step to ensure traceability.

Data Transformation and Modeling

• Construction of a reproducible transformation pipeline (reading, processing, export) with dedicated modular functions (text cleaning, image validation).
• Design of a conceptual data schema (Mermaid) distinguishing between business conceptual model and technical storage schema.
• Documentation of data fields, types, and roles in the AI use case (classification, NLP).

ETL Orchestration with Apache Airflow

• Design of an Airflow DAG structuring extraction, transformation, and loading into distinct and modular tasks.
• Reuse of functions developed in previous steps within Airflow operators (PythonOperator).
• Consideration of the choice and security of the target database (authentication, role management, encryption of sensitive data).

Monitoring and KPI-driven Management

• Definition of performance indicators covering data accuracy, execution speed, and resource cost.
• Construction of a monitoring dashboard (Streamlit) readable by a non-technical audience.
• Formalization of a monitoring plan: alert thresholds, error management, frequency of checks, consistency with existing automations.

Architecture

       Multimodal Sources
  (APIs, RSS, open data, networks)
                │
                ▼
     Automated Extraction
   (modular Python scripts)
                │
                ▼
          Raw Data
      (linked text + image)
                │
                ▼
   Transformation & Cleaning
  (reproducible pipeline + logs)
                │
                ▼
    Conceptual Data Schema
                │
                ▼
       Airflow Orchestration
    (DAG : extract → transform → load)
                │
                ▼
     Secure Database
        (SQL / NoSQL)
                │
                ▼
   KPI Dashboard (Streamlit)
     + Monitoring Plan

Technical Stack

Extraction & Scraping

• Requests, BeautifulSoup, Scrapy, Selenium
• Feedparser (RSS feeds)
• APIs (News API, Reddit, NewsData.io)

Transformation & Modeling

• Python (modular functions, logging)
• Mermaid / draw.io (conceptual schema)

Orchestration

• Apache Airflow (DAG, PythonOperator)

Monitoring & Reporting

• Streamlit (KPI dashboard)

Deliverables

• Source exploration report (Markdown/PDF)
• Extraction scripts (.py / .ipynb)
• Transformation pipeline (.py / .ipynb)
• Executable Airflow DAG with execution proofs
• KPI dashboard + monitoring plan

Best Practices Applied

• Prioritization of official access channels before any scraping
• Systematic verification of usage rights and label reliability
• Clear separation between exploration, extraction, transformation, and orchestration
• Code structured into modular, independently testable functions
• Systematic logging of transformations for traceability
• Securing the target database (authentication, roles, encryption)
• KPI reporting adapted for a non-technical audience

Results

Upon completion of this project, the pipeline allows to:

• automatically extract multimodal publications (text + image) from multiple sources, without manual intervention;
• transform and structure this data according to a documented conceptual schema;
• orchestrate the entire ETL flow via a reproducible Airflow DAG;
• load the data into a secure database, ready for AI model training;
• continuously monitor the quality and performance of the pipeline via a dedicated dashboard.

Skills Acquired

• Sourcing and qualification of multimodal data
• Web scraping and API integration (Requests, BeautifulSoup, Scrapy, Selenium)
• Design of reproducible and logged transformation pipelines
• Conceptual data modeling (Mermaid)
• ETL orchestration with Apache Airflow
• KPI definition and dashboard construction (Streamlit)
• Database security and governance