Docker best-practices in Apache Spark application deployments

Spark-provided container image build
Kubernetes operator container image build

Spark on Kubernetes

Image credits: https://spark.apache.org/docs/latest/img/k8s-cluster-mode.png
Image credits: Spark Docs https://spark.apache.org/docs/latest/running-on-kubernetes.html

Multi-stage builds

For Scala/Java Applications
For PySpark Applications

Testing

Scala Spark Test Application
PySpark Test Application

Next Steps

Summary

--

--

--

CEO & Principal ML/Data Engineer @ data-max.io

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

aelf Enterprise v1.0.0 RC 1 Officially Released

Security Test Strategy for shift left security testing during the Software Development Lifecycle

six stages of the software security lifecycle detailing their security activities

Found 18K+ games on NVIDIA GeForce NOW

Standby-Power-Free Integrated Circuits Using MTJ-Based VLSI Computing for IoT Applications

Integrating Splunk with MuleSoft

What’s new in Visual Studio 2022 and C# 10.0?

visual studio 2022

Bazel Hermetic Toolchain and Tooling Migration

Re-evaluating process and workflow in software development

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sadik Bakiu

Sadik Bakiu

CEO & Principal ML/Data Engineer @ data-max.io

More from Medium

Kafka on GCP Cluster

Apache Kafka on Kubernetes using Strimzi

Strimzi and Kafka

Security & SSL Setup in Confluent Kafka

Scaling Airflow Workers in EKS