Advanced: Kubernetes (K8s) deployment#

Kubernetes (K8s) is an open-source system for automating the deployment, scaling, and management of containerized applications. It is highly suitable for production environments where scalability, reliability, and efficient resource management are crucial.

What is a K8s Deployment?#

A Kubernetes deployment involves running an application within a cluster of nodes, where each node can run multiple containers. This setup provides enhanced capabilities such as automated scaling, self-healing, and easy management of complex applications.

A Syft Server K8s deployment is ideal for:

  • Production Environments: Designed to handle large-scale, production-grade deployments.

  • Large data scale: Working with large quantities of data that require scale computation.

  • Custom Workloads: Dynamically creating Docker images and containers to scale computation.

  • Complex Networking: Utilizing reverse tunneling, reverse proxies, and advanced networking configurations.

  • Observability and Orchestration: Leveraging Helm charts for deployment and upgrades, and using OpenTelemetry for observability.

When do I need this?#

You should consider a K8s deployment if:

  • You require a fully-featured deployment

  • You need to manage and scale your application across multiple nodes.

  • You want to take advantage of advanced features like automated deployments and upgrades, scaling, and observability.

  • You aim to facilitate collaboration with external researchers and require a robust, scalable infrastructure.

  • You do not have cloud resources and want to benefit from the remote tunneling capability to allow externals to reach your node. This is possible via local K3d alone

Steps#

The following tutorials are available to guide you through the K8s deployment process:

  1. Ensure you have access to a local machine/VM for testing with k3d with the necessary requirements and a cloud environment for the k8s deployment

  2. Testing: Deploying locally using K3d

  3. Installing and upgrading your server using Helm Charts

  4. Cloud-specific deployments:

    • Azure: Deploy on Azure (More coming soon)

    • GCP: Deploy on GCP (More coming soon)

    • EKS: Deploy on EKS (More coming soon) Please refer to this tutorial in the meantime

What is a full-stack K8s deployment made of?#

A full-stack K8s deployment includes the following components:

  • Database: Utilizes MongoDB for server data storage and SeeweedFS for large blob data storage.

  • Custom Workloads: Uses Kaniko to dynamically create Docker images and new containers for scalable computation.

  • Networking: Supports reverse tunneling with Rathole and allows configuring Traefik or Ingress as a reverse proxy.

  • Orchestration & Observability: Facilitates easy deployment and upgrades via Helm charts, with observability powered by OpenTelemetry.

  • Frontend: Hosts a web client for the server (currently under development).

#

On this page