RetroFed Forum

Welcome to the nostalgia-filled piefed instance of yesteryear.

Open Registration with Application questionnaire.

If you have a question or comment you can post in !meta@retrofed.com or message @DosDude@retrofed.com or @Die4Ever@retrofed.com

In case of downtime and/or emergency, we will post updates on @DosDude@mastodon.social

Some rules we try to uphold:

Be civil. Bigotry is not welcome. This includes (but is not limited to): transphobia, queerphobia, racism, ageism and ableism. We do not promote hate.
Communities need to be Retro or Retro-adjacent.
Please don't upload pornographic content. (use an external image host for that)

hahattpro@programming.dev Joined: 2023-11-07

The problem is I am using beam.ParDo to call api, to enrich my data.

pipeline | "read_csv" >> beam.io.ReadFromCsv("...") 
                | "get_info_api" >> beam.ParDo(GetApiDoFN()) 
                | "write_output" >>  beam.io.WriteToBigQuery(...)

The problem is that api have limit. I need a way to control so it do not overload api or get 429 error.

I would like to set api call should not exceed 4 call / second.

How should I do it ?

Welcome to the nostalgia-filled piefed instance of yesteryear.

Open Registration with Application questionnaire.

If you have a question or comment you can post in !meta@retrofed.com or message @DosDude@retrofed.com or @Die4Ever@retrofed.com

In case of downtime and/or emergency, we will post updates on @DosDude@mastodon.social

Some rules we try to uphold:

Be civil. Bigotry is not welcome. This includes (but is not limited to): transphobia, queerphobia, racism, ageism and ableism. We do not promote hate.
Communities need to be Retro or Retro-adjacent.
Please don't upload pornographic content. (use an external image host for that)