This blog was written together Martin Rusnak from rusnakconsulting.com and Bujar Bakiu. Not that long ago (maybe somewhere this is still the case), companies had multiple one dimensional data teams. Each team was composed of either only Data Scientists, Business Analysts, or Data Engineers. With this set up, companies struggled to integrate data products into their wider software architecture. …

This post is about Kubeflow, Spark and their interaction. One year ago, I was trying to trigger a Spark job from a Kubeflow pipeline. It was proving to be a stubborn but exciting problem. The other weekend, I started tinkering with this problem again. In the end, persistence and patience…

Apache Spark has become the de facto standard tool for distributed data processing. Similarly, Kubernetes has become the de facto standard container orchestration tool. Deploying Spark jobs on Kubernetes has been natively supported since version 2.3. An alternative to such deployment, is available using operator pattern, see spark-on-k8s-operator. Both these…

There are a lot of platforms and providers out there offering great tools and solutions falling in the scope of an Analytics Platform. Many of them requiring zero manual effort to set them up. But sometimes what is missing is just that little bit of customization to fit our needs…

A big challenge businesses faces is the deployment of machine learning models in production environments. It requires dealing with a complex set of moving parts through different pipelines. Once the models are developed, they need to be trained, deployed, monitored and kept track of. This post tries to describe, how…