Transformation and Modelling

We will use dbt for any processig and modelling that will be done to the data. When we curate the data, we will move the curated to an analytical database.

Clickhouse

We will be using clickhouse for the Analytics database. What I have learned is that the OSS does not have an in-built UI but you can use third-party app for this.

TODO: Fine-tuning Clickhouse for larger datasets

DuckDB

We will use Duckdb and Iceberg connected to S3. This is an experiment that can change after the first project trial.