ML Orchestration Made Easy with Apache Liminal and Amazon MWAA
by Lior Schachter & Lidor Ettinger The challenges involved in operationalizing machine learning models are one of the main reasons
May 8, 2022
No Comments
Lidorettinger
Autoscaling your Airflow using DataDog External Metrics
Follow this step-by-step guide and learn how to autoscale your Airflow K8s pods based on ExternalMetrics One of the strengths
January 16, 2022
No Comments
First Steps to Microservices — Natural Intelligence
At Natural Intelligence we are running a Kubernetes cluster on AWS which makes it is easy to deploy, manage and
September 24, 2020
No Comments
Monitoring Confluent Cloud Kafka with Datadog — Natural Intelligence
When I came across Apache Kafka and its concept of a streaming platform I asked myself — how will I monitor it?
July 16, 2020
No Comments
Using AWS SSM As A Unified Credentials Storage
At Natural Intelligence, we have plenty of dockerized services that run on kubernetes and swarm cluster on AWS. Each service
May 7, 2020
No Comments
Custom GitHub Checks With Jenkins Pipeline
At Natural Intelligence, we are bringing massive complex pipelines into production and want to support multiple teams and their orchestration
March 26, 2020
No Comments
CI/CD DAGs Pipeline Deployment On Kubernetes
In the previous blog we talked about how to deploy Airflow DAGs, handle task processing at scale and how to
February 2, 2020
No Comments
Key Airflow Pipeline Deployment Lessons On Kubernetes
At Natural Intelligence we use Airflow extensively in order to create simple ETL flows as well as complex ML tasks
November 28, 2019
No Comments