Mirrord Mirrord on the wall, who's most processed of them all 🪞🔄

Exploring Mirrord 1: The quickest process mirroring from Local to Kubernetes

In today’s post, we are going to explore mirrord. Mirrord is a process mirroring tool which mirrors your local running process and related dependencies to a Kuebrnetes cluster. We will see how to quickly get started with mirrord, and how it works.

31st October 2024

This will take a minute.

Installing Mirrord CLI on the Local system

I am currently operating a M2 Mac so running the command below to install the mirrord CLI.

brew install metalbear-co/mirrord/mirrord

We need a service running in the Kubernetes cluster against which we can test our process mirroring. We will be sending a HTTP GET request to the echo-server through process mirroring.

Installing Echo Server on Kubernetes

Start minikube.

minikube start

Let’s install the echo-server with the following command.

helm repo add ealenn https://ealenn.github.io/charts
helm repo update
helm install echo-server ealenn/echo-server --namespace echo-server --force --create-namespace 

Run the command below to send HTTP GET request to the echo-server ClusterIP Service.

mirrord exec curl echo-server

The echo-server Service is only available from within the context of the Kubernetes cluster (and namespace). The command above runs curl echo-server in the context the Kubernetes cluster and we get the output below.

* Running binary "/var/folders/v7/yqyq_d6x2996hfnnwvgs5f080000gn/T/mirrord-bin-ghu3278mz/usr/bin/curl" with arguments: ["echo-server"].
* mirrord will run without a target, no configuration file was loaded
* operator: the operator will be used if possible
* env: all environment variables will be fetched
* fs: file operations will default to read only from the remote
* incoming: incoming traffic will be mirrored
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
⠁ mirrord exec
    ✓ running on latest (3.122.1)!
    ✓ ready to launch process
      ✓ layer extracted
      ✓ operator not found
      ✓ agent pod created
      ✓ pod is ready
      ✓ arm64 layer library extracted
    ✓ config summary                                                                                    
{"host":{"hostname":"echo-server","ip":"::ffff:10.244.0.10","ips":[]},"http":{"method":"GET","baseUrl":"","originalUrl":"/","protocol":"http"},"request":{"params":{"0":"/"},"query":{},"cookies":{},"body":{},"headers":{"host":"echo-server","user-agent":"curl/8.4.0","accept":"*/*"}},"environment":{"PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","HOSTNAME":"echo-server-cbc7ddb7b-s9c6q","ENABLE__COOKIES":"true","ENABLE__HTTP":"true","ENABLE__HEADER":"true","ENABLE__HOST":"true","ENABLE__REQUEST":"true","LOGS__IGNORE__PING":"false","PORT":"80","ENABLE__ENVIRONMENT":"true","ENABLE__FILE":"true","KUBERNETES_PORT_443_TCP_ADDR":"10.96.0.1","ECHO_SERVER_SERVICE_HOST":"10.103.113.87","ECHO_SERVER_PORT_80_TCP":"tcp://10.103.113.87:80","ECHO_SERVER_PORT_80_TCP_ADDR":"10.103.113.87","KUBERNETES_SERVICE_PORT":"443","KUBERNETES_SERVICE_PORT_HTTPS":"443","KUBERNETES_PORT_443_TCP_PROTO":"tcp","KUBERNETES_PORT_443_TCP_PORT":"443","KUBERNETES_SERVICE_HOST":"10.96.0.1","KUBERNETES_PORT_443_TCP":"tcp://10.96.0.1:443","ECHO_SERVER_SERVICE_PORT":"80","ECHO_SERVER_SERVICE_PORT_HTTP":"80","ECHO_SERVER_PORT_80_TCP_PROTO":"tcp","ECHO_SERVER_PORT_80_TCP_PORT":"80","KUBERNETES_PORT":"tcp://10.96.0.1:443","ECHO_SERVER_PORT":"tcp://10.103.113.87:80","NODE_VERSION":"16.16.0","YARN_VERSION":"1.22.19","HOME":"/root"}}%

The last part of the output after “config summary” was returned from the echo-server Service.

The Mirrord Agent (Targetless mode)

The mirrord-agent acts as a proxy to the local process due to which it can run in the context of the Kubernetes cluster. More on the agent here. When we run `mirrord exec curl echo-server` the CLI basically creates a Job called the mirrord-agent which looks like the one below.

k get job                                                                                          
NAME                       COMPLETIONS   DURATION   AGE
mirrord-agent-95u3z3q18t   0/1           6s         6s

And below are the logs for the agent. Had to quickly run it as the exec command runs.

kubectl logs -f job/$(kubectl get jobs --output=jsonpath='{.items[*].metadata.name}')

agent ready - version 3.122.1
  2024-10-30T22:21:14.035109Z  WARN mirrord_agent::outgoing::udp: interceptor_task -> no messages left
    at mirrord/agent/src/outgoing/udp.rs:225 on ThreadId(5)

  2024-10-30T22:21:19.172187Z  WARN mirrord_agent::outgoing::udp: interceptor_task -> no messages left
    at mirrord/agent/src/outgoing/udp.rs:225 on ThreadId(7)

  2024-10-30T22:21:19.172908Z  INFO mirrord_agent::entrypoint: main -> mirrord-agent `start` exiting successfully.
    at mirrord/agent/src/entrypoint.rs:824 on ThreadId(1)

We don’t have any critical information which we can work with here for sure except an observation that there is no UDP data available which can be forwarded to the user on the local host, hence the warning messages above. Apart from that the agent process is exiting successfully.

Let’s take apart the Job spec now. Let’s run the command below and stare at the spec for a bit.

kubectl edit job/$(kubectl get jobs --output=jsonpath='{.items[*].metadata.name}') 

We get the following output. I have annotated important/relevant lines,

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    batch.kubernetes.io/job-tracking: ""
    linkerd.io/inject: disabled
    sidecar.istio.io/inject: "false"
  creationTimestamp: "2024-10-30T22:32:02Z"
  generation: 1
  labels:
    app: mirrord
    kuma.io/sidecar-injection: disabled ###
  name: mirrord-agent-q4xcvitguv
  namespace: echo-server
  resourceVersion: "7817"
  uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3
spec:
  backoffLimit: 0
  completionMode: NonIndexed
  completions: 1
  parallelism: 1
  selector:
    matchLabels:
      batch.kubernetes.io/controller-uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3
 suspend: false
  template:
    metadata:
      annotations:
        linkerd.io/inject: disabled  ###
        sidecar.istio.io/inject: "false" ###
      creationTimestamp: null
      labels:
        app: mirrord
        batch.kubernetes.io/controller-uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3
        batch.kubernetes.io/job-name: mirrord-agent-q4xcvitguv
        controller-uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3
        job-name: mirrord-agent-q4xcvitguv
        kuma.io/sidecar-injection: disabled
    spec:
      containers:
      - command:
        - ./mirrord-agent
        - -l
        - "33863"
        - targetless ###
        env:
        - name: RUST_LOG
          value: info
        - name: MIRRORD_AGENT_STEALER_FLUSH_CONNECTIONS ##
          value: "true"
        - name: MIRRORD_AGENT_NFTABLES
          value: "false"
        - name: MIRRORD_AGENT_JSON_LOG
          value: "false"
        image: ghcr.io/metalbear-co/mirrord:3.122.1
        imagePullPolicy: IfNotPresent
        name: mirrord-agent
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 1m
            memory: 1Mi
        securityContext:
          privileged: false ###
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      tolerations:
      - operator: Exists
  ttlSecondsAfterFinished: 1
status:
  active: 1
  ready: 1
  startTime: "2024-10-30T22:32:02Z"
  uncountedTerminatedPods: {}

Sidecar injection disabled

Based on the labels and annotations given above we can tell that these have the Kuma, Istio and LinkerD sidecar injection disabled.

Targetless

Targetless is the mode because of which we were able to run `curl echo-server` in the content of the Kubernetes cluster without specifying a particular target pod or container. The opposite of this,l targets a particular pod and intercepts/mirrors the traffic and environment of the process in the pod (Targeted mode).

Flush existing connections

MIRRORD_AGENT_STEALER_FLUSH_CONNECTIONS is an environment variable is set to true which means that existing connections are flushed for more accurate interception of traffic.

Unprivileged Container

The agent container runs in an unprivileged mode but needs the below Linux capabilities to act as a proxy for processes. These capabilities in Linux allow granular control to be given to processes based on what they can or cannot do. We can see these being used in the targeted mode and not in the targetless mode we were using right now.

• CAP_NET_ADMIN and CAP_NET_RAW - used for modifying routing tables.

  • CAP_NET_ADMIN allows network interface, firewall, routing tables, socket permissions configurations etc.

  • CAP_NET_RAW allows for creation of RAW and PACKET socket types.

• CAP_SYS_PTRACE - used for reading target pod environment.

  • Gives the ability to use the `ptrace` system call which is usually used for debugging and process control and in this case we can use it to get information about the process running in the Pod.

• CAP_SYS_ADMIN - used for joining target pod network namespace.

  • Gives the ability to mount filesystems, change system time, manage devices etc.

(this discussion is continued in the next batch of relevant diary entries/post …)