Architecture

This document describes the technical architecture of the Cisco Virtual Kubelet Provider.

Overview

The Cisco Virtual Kubelet Provider implements the Virtual Kubelet provider interface, enabling Kubernetes to treat Cisco IOS-XE devices as compute nodes for container workloads.

Component Architecture

┌─────────────────────────────────────────────────────────────────-─────┐
│                         Kubernetes Cluster                            │
│  ┌────────────────────────────────────────────────────────────────┐   │
│  │                      API Server                                │   │
│  └─────────────────────────────┬──────────────────────────────────┘   │
│                                │                                      │
│  ┌─────────────────────────────┴──────────────────────────────────┐   │
│  │                    Virtual Kubelet Library                     │   │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐  │   │
│  │  │    Node      │  │     Pod      │  │   AppHostingProvider │  │   │
│  │  │  Controller  │  │  Controller  │  │   AppHostingNode     │  │   │
│  │  └──────────────┘  └──────────────┘  └──────────┬───────────┘  │   │
│  │                                                  │             │   │
│  │  ┌───────────────────────────────────────────────┴───────────┐ │   │
│  │  │                    Driver Layer                           │ │   │
│  │  │  ┌─────────────────────────────────────────────────────┐  │ │   │
│  │  │  │              XEDriver (IOS-XE)                      │  │ │   │
│  │  │  │  ┌─────────────┐ ┌─────────────┐ ┌───────────────┐  │  │ │   │
│  │  │  │  │ App Hosting │ │    Pod      │ │   RESTCONF    │  │  │  │  │
│  │  │  │  │  Lifecycle  │ │  Lifecycle  │ │    Client     │  │  │  │  │
│  │  │  │  └─────────────┘ └─────────────┘ └───────────────┘  │  │  │  │
│  │  │  └─────────────────────────────────────────────────────┘  │  │  │
│  │  └───────────────────────────────────────────────────────────┘  │  │
│  └─────────────────────────────────────────────────────────────────┘  │
└───────────────────────────────────────────────────────────────────────┘
                                    │
                                    │ RESTCONF/HTTPS
                                    ▼
┌───────────────────────────────────────────────────────────────────────┐
│                     Cisco Catalyst 8000V (IOS-XE)                     │
│  ┌─────────────────────────────────────────────────────────────────┐  │
│  │                       IOx Platform                              │  │
│  │  ┌───────────────┐  ┌───────────────┐  ┌───────────────────┐    │  │
│  │  │  Container 1  │  │  Container 2  │  │   Container N     │    │  │
│  │  │   (app)       │  │   (app)       │  │   (app)           │    │  │
│  │  └───────────────┘  └───────────────┘  └───────────────────┘    │  │
│  │                           │                                     │  │
│  │  ┌────────────────────────┴─────────────────────────────────-─┐ │  │
│  │  │              VirtualPortGroup0 + DHCP Pool                 │ │  │
│  │  └────────────────────────────────────────────────────────────┘ │  │
│  └─────────────────────────────────────────────────────────────────┘  │
└───────────────────────────────────────────────────────────────────────┘

Core Components

AppHostingProvider

The main provider struct that implements the Virtual Kubelet nodeutil.Provider interface:

// internal/provider/provider.go
type AppHostingProvider struct {
    ctx             context.Context
    deviceSpec      *v1alpha1.DeviceSpec
    driver          drivers.CiscoKubernetesDeviceDriver
    podsLister      corev1listers.PodLister
    configMapLister corev1listers.ConfigMapLister
    secretLister    corev1listers.SecretLister
    serviceLister   corev1listers.ServiceLister
}

Implemented Interface Methods:

CreatePod(ctx, pod) - Deploy container to device
UpdatePod(ctx, pod) - Update container configuration
DeletePod(ctx, pod) - Remove container from device
GetPod(ctx, namespace, name) - Get pod with status
GetPodStatus(ctx, namespace, name) - Get pod status only
GetPods(ctx) - List all pods on device

AppHostingNode

Implements the node.NodeProvider interface for node heartbeat management:

// internal/provider/provider.go
type AppHostingNode struct{}

func (a *AppHostingNode) Ping(ctx context.Context) error
func (a *AppHostingNode) NotifyNodeStatus(ctx context.Context, cb func(*v1.Node))

Driver Factory

The driver factory pattern allows extensible device support:

// internal/drivers/factory.go
func NewDriver(ctx context.Context, spec *v1alpha1.DeviceSpec) (CiscoKubernetesDeviceDriver, error) {
    switch spec.Driver {
    case v1alpha1.DeviceDriverFAKE:
        return fake.NewAppHostingDriver(ctx, spec)
    case v1alpha1.DeviceDriverXE:
        return iosxe.NewAppHostingDriver(ctx, spec)
    case v1alpha1.DeviceDriverXR:
        return nil, fmt.Errorf("unsupported device type")
    default:
        return nil, fmt.Errorf("unsupported device type")
    }
}

type CiscoKubernetesDeviceDriver interface {
    GetDeviceResources(ctx context.Context) (*v1.ResourceList, error)
    DeployPod(ctx context.Context, pod *v1.Pod) error
    UpdatePod(ctx context.Context, pod *v1.Pod) error
    DeletePod(ctx context.Context, pod *v1.Pod) error
    GetPodStatus(ctx context.Context, pod *v1.Pod) (*v1.Pod, error)
    ListPods(ctx context.Context) ([]*v1.Pod, error)
}

XEDriver (IOS-XE Driver)

Implements the device driver for Cisco IOS-XE app-hosting:

// internal/drivers/iosxe/driver.go
type XEDriver struct {
    config       *v1alpha1.DeviceSpec
    client       common.NetworkClient
    marshaller   func(any) ([]byte, error)
    unmarshaller UnmarshalFunc
}

Key Methods:

CheckConnection(ctx) - Validate device connectivity
GetDeviceResources(ctx) - Report available resources
DeployPod(ctx, pod) - Full pod deployment lifecycle
DeletePod(ctx, pod) - Full pod deletion lifecycle

RestconfClient

HTTP client for RESTCONF API communication:

// internal/drivers/common/restconf_client.go
type RestconfClient struct {
    BaseURL    string
    HTTPClient *http.Client
    Username   string
    Password   string
}

func (c *RestconfClient) Get(ctx, path, result, unmarshal) error
func (c *RestconfClient) Post(ctx, path, payload, marshal) error
func (c *RestconfClient) Patch(ctx, path, payload, marshal) error
func (c *RestconfClient) Delete(ctx, path) error

API Communication

RESTCONF Endpoints

Operation	Method	Endpoint
Configure App	POST	`/restconf/data/Cisco-IOS-XE-app-hosting-cfg:app-hosting-cfg-data/apps`
Install	POST	`/restconf/operations/Cisco-IOS-XE-rpc:app-hosting`
Activate	POST	`/restconf/operations/Cisco-IOS-XE-rpc:app-hosting`
Start	POST	`/restconf/operations/Cisco-IOS-XE-rpc:app-hosting`
Stop	POST	`/restconf/operations/Cisco-IOS-XE-rpc:app-hosting`
Deactivate	POST	`/restconf/operations/Cisco-IOS-XE-rpc:app-hosting`
Uninstall	POST	`/restconf/operations/Cisco-IOS-XE-rpc:app-hosting`
Delete Config	DELETE	`/restconf/data/Cisco-IOS-XE-app-hosting-cfg:app-hosting-cfg-data/apps/app={appID}`
Get State	GET	`/restconf/data/Cisco-IOS-XE-app-hosting-oper:app-hosting-oper-data`
Get ARP	GET	`/restconf/data/Cisco-IOS-XE-arp-oper:arp-data`

YANG Models Used

Cisco-IOS-XE-app-hosting-cfg.yang - App-hosting configuration
Cisco-IOS-XE-app-hosting-oper.yang - App-hosting operational state
Cisco-IOS-XE-rpc.yang - RPC operations (install, activate, start, etc.)
Cisco-IOS-XE-arp-oper.yang - ARP table for IP discovery

Data Flow

Pod Creation Flow

1. kubectl apply -f pod.yaml
         │
         ▼
2. Kubernetes API Server
         │
         ▼
3. Virtual Kubelet Pod Controller
         │
         ▼
4. AppHostingProvider.CreatePod()
         │
         ▼
5. XEDriver.DeployPod()
         │
         ▼
6. XEDriver.CreatePodApps()
         │
         ├─► Configure app-hosting (RESTCONF POST)
         │   - Set VirtualPortGroup interface
         │   - Set resource profile (CPU, memory, disk)
         │   - Set container labels for discovery
         │
         ├─► InstallApp (RESTCONF RPC: install)
         │
         ├─► WaitForAppStatus("DEPLOYED")
         │
         ├─► ActivateApp (RESTCONF RPC: activate)
         │
         ├─► WaitForAppStatus("ACTIVATED")
         │
         └─► Start is automatic (start: true in config)
         │
         ▼
7. Container receives DHCP IP from device pool
         │
         ▼
8. Pod status updated via GetPodStatus()
   - IP discovered from oper-data or ARP table

Pod Deletion Flow

1. kubectl delete pod <name>
         │
         ▼
2. AppHostingProvider.DeletePod()
         │
         ▼
3. XEDriver.DeletePod()
         │
         ▼
4. For each container:
         │
         ├─► StopApp (RESTCONF RPC: stop)
         │
         ├─► WaitForAppStatus("ACTIVATED")
         │
         ├─► DeactivateApp (RESTCONF RPC: deactivate)
         │
         ├─► WaitForAppStatus("DEPLOYED")
         │
         ├─► UninstallApp (RESTCONF RPC: uninstall)
         │
         ├─► WaitForAppNotPresent()
         │
         └─► Delete config (RESTCONF DELETE)

Pod Status Discovery

The provider discovers pod status by:

Container Discovery: Query app-hosting config for apps with matching pod UID labels
State Mapping: Map IOS-XE app states to Kubernetes container states
IP Discovery:
- First, check app-hosting-oper-data for IPv4 address
- Fallback: Look up container MAC address in ARP table

State Management

Pod State Mapping

Kubernetes Phase	IOS-XE App State
Pending	INSTALLING, DEPLOYED, ACTIVATED
Running	RUNNING
Succeeded	STOPPED
Failed	ERROR

Container Labels

Containers are tagged with labels in the --run-opts for discovery:

--label cisco.vk/pod-name=<pod-name>
--label cisco.vk/pod-namespace=<namespace>
--label cisco.vk/pod-uid=<uid>
--label cisco.vk/container-name=<container-name>

App Naming Convention

App IDs are generated from pod metadata:

{pod-uid-without-hyphens}-{container-name-hash}

Networking

DHCP Mode

When dhcpEnabled: true:

App-hosting is configured with only the VirtualPortGroup interface number
Container requests IP from device DHCP pool
Provider discovers IP from:
- app-hosting-oper-data network interfaces (preferred)
- ARP table lookup using container MAC address (fallback)

Network Configuration Flow

┌─────────────────────────────────────────────────────────────────┐
│                    Catalyst 8000V                                │
│                                                                  │
│  ┌────────────┐    ┌─────────────────┐    ┌─────────────────┐  │
│  │ Container  │───►│ VirtualPortGroup0│───►│   DHCP Pool     │  │
│  │ (eth0)     │    │  192.168.1.254        │   192.168.1.0/24│  │
│  │            │◄───│  (gateway)      │◄───│   assigns IP    │  │
│  └────────────┘    └─────────────────┘    └─────────────────┘  │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Project Structure

cisco-virtual-kubelet/
├── api/v1alpha1/                 # CRD-ready API types
│   ├── doc.go
│   ├── groupversion_info.go
│   ├── types.go                  # DeviceSpec, CiscoDevice, shared types
│   └── xe_types.go               # IOS-XE driver-specific types
├── cmd/virtual-kubelet/          # Entry point
│   ├── main.go                   # Main function
│   └── root.go                   # CLI setup with cobra
├── internal/
│   ├── config/                   # Configuration
│   │   └── config.go             # YAML/viper loader → DeviceSpec
│   ├── provider/                 # VK Provider
│   │   ├── provider.go           # AppHostingProvider
│   │   └── defaults.go           # Node defaults
│   └── drivers/                  # Device drivers
│       ├── factory.go            # Driver factory
│       ├── common/               # Shared code
│       │   ├── restconf_client.go
│       │   ├── types.go
│       │   ├── naming.go
│       │   └── helpers.go
│       ├── iosxe/                # IOS-XE driver
│       │   ├── driver.go         # XEDriver
│       │   ├── app_hosting.go    # App lifecycle
│       │   ├── pod_lifecycle.go  # Pod CRUD
│       │   ├── transformers.go   # K8s ↔ IOS-XE
│       │   └── models.go         # YANG structs
│       └── fake/                 # Test driver
│           └── driver.go
└── dev/                          # Development files

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Architecture

Overview

Component Architecture

Core Components

AppHostingProvider

AppHostingNode

Driver Factory

XEDriver (IOS-XE Driver)

RestconfClient

API Communication

RESTCONF Endpoints

YANG Models Used

Data Flow

Pod Creation Flow

Pod Deletion Flow

Pod Status Discovery

State Management

Pod State Mapping

Container Labels

App Naming Convention

Networking

DHCP Mode

Network Configuration Flow

Project Structure

FilesExpand file tree

ARCHITECTURE.md

Latest commit

History

ARCHITECTURE.md

File metadata and controls

Architecture

Overview

Component Architecture

Core Components

AppHostingProvider

AppHostingNode

Driver Factory

XEDriver (IOS-XE Driver)

RestconfClient

API Communication

RESTCONF Endpoints

YANG Models Used

Data Flow

Pod Creation Flow

Pod Deletion Flow

Pod Status Discovery

State Management

Pod State Mapping

Container Labels

App Naming Convention

Networking

DHCP Mode

Network Configuration Flow

Project Structure