RetroFed Forum
Your home away from modern life.
Search Categories Forum list Register Log in
About RetroFed
RetroFed

Welcome to the nostalgia-filled piefed instance of yesteryear.

Open Registration with Application questionnaire.

If you have a question or comment you can post in !meta@retrofed.com or message @DosDude@retrofed.com or @Die4Ever@retrofed.com

In case of downtime and/or emergency, we will post updates on @DosDude@mastodon.social

Some rules we try to uphold:

  • Be civil. Bigotry is not welcome. This includes (but is not limited to): transphobia, queerphobia, racism, ageism and ableism. We do not promote hate.
  • Communities need to be Retro or Retro-adjacent.
  • Please don't upload pornographic content. (use an external image host for that)
75 registered users
Administrators: DosDude, Die4Ever
Visit retrofed.com
A guide how to adopt an existing Spark scala library for Spark Connect
Post Reply RetroFed Forum Forum Index » Data Engineering
AuthorMessage
Sem lemmy.ml sem@lemmy.ml
Joined: 2023-04-27
Quote

Link: https://semyonsinchenko.github.io/ssinchenko/post/porting_deequ_to_sparkconnect/

Let me share my post with a detailed step by step guide how an exisiting Spark scala library may be adopted to work with recently introduced Spark Connect. As an example I have chosen a pupular open source data quality tool AWS Deequ. I made all the necessary protobuf messages and a Spark Connect Plugin. I tested it from PySpark Connect 3.5.1 and it works. Of course, all the code is public in git.
1   Original
Post Reply
Back to top
Powered by PieFed · PieBB frontend · All times are Europe/Amsterdam