Scala functions, for use in Apache Spark, designed for use with the data-linking python package Splink.
Built packages can be found in the jars folder.
These are not published in any package repository, but relevant ones are bundled with Splink.
The goal is to build the package into a .jar file, which can then be used by Splink.
This is done using maven, via docker. Just run make package.
You can check if it is working with Splink by running make check-splink.
For Spark 3 compatible versions, which need to be compiled against a different version of Scala, set the environment variable SPARK_COMPAT=spark3
For more information check out the dev readme.
This module was started as an extension of an example provided in [1] Phillip Lee (ONS) has created an example of a UDF defined in Scala, callable from PySpark, that wraps a call to JaroWinklerDistance from Apache commons.
See the changelog.
[1]: Using a Scala UDF Example @philip-lee-ons