Apache Spark 3.0 adds Nvidia GPU support for machine learning

Apache Spark, the in-memory large facts processing framework, will turn out to be fully GPU

Apache Spark, the in-memory large facts processing framework, will turn out to be fully GPU accelerated in its shortly-to-be-released three. incarnation. Best of all, today’s Spark applications can choose advantage of the GPU acceleration devoid of modification existing Spark APIs all function as-is.

The GPU acceleration components, offered by Nvidia, are built to complement all phases of Spark applications which includes ETL functions, machine discovering training, and inference serving.

Nvidia’s Spark contributions draw on the RAPIDS suite of GPU-accelerated facts science libraries. Quite a few of RAPIDS’ internal facts constructions, like dataframes, complement Spark’s have, but obtaining Spark to use RAPIDS natively has taken just about 4 yrs of function.

Spark three. speedups really don’t arrive exclusively from GPU acceleration. Spark three. also reaps overall performance gains by reducing facts motion to and from GPUs. When facts does require to be moved throughout a cluster, the Unified Communication X framework shuttles it instantly from a person block of GPU memory to an additional with negligible overhead.

In accordance to Nvidia, a preview launch of Spark three. functioning on the Databricks system yielded a seven-fold overall performance improvement when applying GPU acceleration, although information about the workload and its dataset were not out there. 

No firm day has been presented for typical availability of Spark three.. You can download preview releases from the Apache Spark job site.

Copyright © 2020 IDG Communications, Inc.