Advanced: Kubernetes (K8s) deployment

Advanced: Kubernetes (K8s) deployment

Kubernetes (K8s) is an open-source system for automating the deployment, scaling, and management of containerized applications. It is highly suitable for production environments where scalability, reliability, and efficient resource management are crucial.

What is a K8s Deployment?

A Kubernetes deployment involves running an application within a cluster of nodes, where each node can run multiple containers. This setup provides enhanced capabilities such as automated scaling, self-healing, and easy management of complex applications.

A Syft Server K8s deployment is ideal for:

  • Production Environments: Designed to handle large-scale, production-grade deployments.

  • Large data scale: Working with large quantities of data that require scale computation.

  • Custom Workloads: Dynamically creating Docker images and containers to scale computation.

  • Complex Networking: Utilizing reverse tunneling, reverse proxies, and advanced networking configurations.

  • Observability and Orchestration: Leveraging Helm charts for deployment and upgrades, and using OpenTelemetry for observability.

When do I need this?

You should consider a K8s deployment if:

  • You require a fully-featured deployment

  • You need to manage and scale your application across multiple nodes.

  • You want to take advantage of advanced features like automated deployments and upgrades, scaling, and observability.

  • You aim to facilitate collaboration with external researchers and require a robust, scalable infrastructure.

  • You do not have cloud resources and want to benefit from the remote tunneling capability to allow externals to reach your node. This is possible via local K3d alone

Steps

The following tutorials are available to guide you through the K8s deployment process:

  1. Ensure you have access to a local machine/VM for testing with k3d with the necessary requirements and a cloud environment for the k8s deployment

  2. Testing: Deploying locally using K3d

  3. Installing and upgrading your server using Helm Charts

  4. Cloud-specific deployments:

    • Azure: Deploy on Azure (More coming soon)

    • GCP: Deploy on GCP (More coming soon)

    • EKS: Deploy on EKS (More coming soon) Please refer to this tutorial in the meantime

What is a full-stack K8s deployment made of?

A full-stack K8s deployment includes the following components:

  • Database: Utilizes MongoDB for server data storage and SeeweedFS for large blob data storage.

  • Custom Workloads: Uses Kaniko to dynamically create Docker images and new containers for scalable computation.

  • Networking: Supports reverse tunneling with Rathole and allows configuring Traefik or Ingress as a reverse proxy.

  • Orchestration & Observability: Facilitates easy deployment and upgrades via Helm charts, with observability powered by OpenTelemetry.

  • Frontend: Hosts a web client for the server (currently under development).

On This Page