RetroFed Forum

Welcome to the nostalgia-filled piefed instance of yesteryear.

Open Registration with Application questionnaire.

If you have a question or comment you can post in !meta@retrofed.com or message @DosDude@retrofed.com or @Die4Ever@retrofed.com

In case of downtime and/or emergency, we will post updates on @DosDude@mastodon.social

Some rules we try to uphold:

Be civil. Bigotry is not welcome. This includes (but is not limited to): transphobia, queerphobia, racism, ageism and ableism. We do not promote hate.
Communities need to be Retro or Retro-adjacent.
Please don't upload pornographic content. (use an external image host for that)

jupyter@programming.dev Joined: 2024-06-22

I am creating a couple of bigger database tables with at least hundreds of millions of observations, but growing. Some tables are by minute, some by milliseconds. timestamps are not necessarily unique.

Should I create separate year, month, or date and time columns? Is one unique datetime column enough? At what size would you partition the tables?

Raw data is in csv.

Currently I aim for postgres and duckdb. Does timescaledb make a significant difference?

Welcome to the nostalgia-filled piefed instance of yesteryear.

Open Registration with Application questionnaire.

If you have a question or comment you can post in !meta@retrofed.com or message @DosDude@retrofed.com or @Die4Ever@retrofed.com

In case of downtime and/or emergency, we will post updates on @DosDude@mastodon.social

Some rules we try to uphold:

Be civil. Bigotry is not welcome. This includes (but is not limited to): transphobia, queerphobia, racism, ageism and ableism. We do not promote hate.
Communities need to be Retro or Retro-adjacent.
Please don't upload pornographic content. (use an external image host for that)