"Let's hook up !" ~ said every LD_PRELOAD ever 🪝⚙️

Exploring Mirrord 2: overriding/hooking into libc functions with LD_PRELOAD + mirrord-layer

In the last entry, I explored mirrord on a very surface level. I mirrored a local process into Kubernetes using the `mirrord exec` command and explored the mirrord-agent to get a basic understanding of the environment in which the mirrored process ran. This entry continues from the last time where we left off in trying to understand the inner workings of mirrord.

If you would like to catch up on the last exploration, feel free to navigate to the previous entry through the link below.

Mirrord Mirrord on the wall, who's most processed of them all 🪞🔄

Targeted Mode in Mirrord

Targetless vs Targeted Mode

2nd November 2024

Last time we ran mirrord in Targetless Mode. We ran a curl command through mirrord for it to get mirrored to the context of a Kubernetes cluster. We were then able to access the ClusterIP echo-server Service which wasn’t exposed.

The environment in which the mirrored process (mirrored by mirrord) runs is made possible by Linux Capabilities. We only scratched the surface last time and didn’t get to exactly see how they manifest themselves in a Pod. We learnt a little about the Linux Capabilities through the documentation, but, we didn’t see them in action.

To see them in action we need to use mirrord in the Targeted Mode. In this case, mirrord is going to run the process locally and then use mirrord-layer to hook the execution of the libc functions. Then it will listen for the traffic/output of the mirrored process in the mirrord-agent and route the information to the mirrord-layer which will pass the information onto the hooked/replaced/detoured (can be used interchangably) libc functions.

Let’s run `ls` with mirrord while defining a target on the Kubernetes cluster. The target here is the process context in which `ls` should run. The first Pod from the given Deployment will be chosen as the target for environment mirroring here. Run the following command.

mirrord exec -t deployment/coredns -n kube-system ls --target-namespace kube-system

We get the below output for the above.

When targeting multi-pod deployments, mirrord impersonates the first pod in the deployment.
Support for multi-pod impersonation requires the mirrord operator, which is part of mirrord for Teams.
You can get started with mirrord for Teams at this link: https://mirrord.dev/docs/overview/teams/?utm_source=multipodwarn&utm_medium=cli
* Running binary "/var/folders/v7/yqyq_d6x2996hfnnwvgs5f080000gn/T/mirrord-bin-ghu3278mz/bin/ls" with arguments: ["/"].
* mirrord will target: deployment/coredns, no configuration file was loaded
* operator: the operator will be used if possible
* env: all environment variables will be fetched
* fs: file operations will default to read only from the remote
* incoming: incoming traffic will be mirrored
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
⠂ mirrord exec
    ✓ running on latest (3.122.1)!
    ✓ ready to launch process
      ✓ layer extracted
      ✓ operator not found
      ✓ agent pod created
      ✓ pod is ready
      ✓ arm64 layer library extracted
    ✓ config summary                                                                                                                 
Applications    System          Volumes         cores           etc             opt             sbin            usr
Library         Users           bin             dev             home            private         tmp             var

If `ls` ran in the context of the given target, why do I still get the output from my Mac filesystem. This seems to be a bit wrong. But it isn’t. Mirrord’s relative context is always local, but the other bits of the context depend on the libc hooks provided by mirrord-layer. That begs the question, what do we even mean by the context here ?

Understanding Context

Context of a process

The context of a process is usually considered as the environment (network, filesystem ) it is run in. The environment also contains the dependencies it requires to run the process + the network context it is being run from. Let’s look at the context as different parameters fed into the process for it to execute. The parameters being the dependencies, filesystem and the network itself. But, if this is the case then the context is not completely altered for the processes running with mirrord and that’s why `ls` did not give an output from the container filesystem.

What I really trying to understand here is why `ls` does not show output from the container instead of the local machine considering that it is being run in the “context” of the container. I realized this was wrong till I figured out that the process is mirrored, it isn’t relocated for running in any way. We are simulating the environment of the container to the local process with mirrord and that’s about it.

Introduction to mirrord-layer

I read this blog post while trying to understand the process context a bit more and it really helped understand that mirrord just runs the process as is locally and hooks into it by intercepting/overriding the libc function calls (I am also not covering anything that is already given in the same blog post). Mirrord intercepts the execution of these calls through another component called the mirrord-layer which picks up on the function call before it’s executed so that when these function calls do get executed they will run the detours/hooks/replacements of the libc calls.

Mirrord preloads this mirrord_layer before any of the libc functions are loaded by using the LD_PRELOAD (on Linux and DYLD_INSERT_LIBRARIES on Mac).

A necessary tangent

Skip this section if you don’t want to see how I failed trying to figure out how to use dtrace with Mac before giving up on it. TLDR: I tried using it, disabled SIP after not wanting to disable it, ran dtrace/dtruss and it hung multiple times.

Tangent: DLLs, Fails with SIP (System Integrity Protection) and dtrace

Dynamically Linked Libraries

Apart from that I just wanted to see which dynamically linked libraries were being loaded onto `ls`. I am using a Mac so instead of `ldd` to check this I used `otool` for checking the dynamically linked libraries.

otool -L /bin/ls                
                                                                                                                                              
/bin/ls:
        /usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)

We can see that instead of `.so` in Linux, Mac OS uses `.dylib`as the extension for dynamically linked libraries. First one’s short for shared objects and the other is short for dynamically linked library.

Mirrord exec execution and Mirrord Configuration

This function runs when `mirrord exec` is collecting the context as parameters, and this is the function which actually executes the given process (in this case `ls`) based on the given contextual parameters. One can provide a configuration with mirrord based on which the mirrord components can make the process mirroring work for your use case. For more on mirrord configuration check this page out.

A dtrace sob story

A System Integrity Protection (SIP) Dilemma

I want to understand how the output of `ls` in this case can be given in the context of the relative filesystem of the mirrord-agent. Let’s run a `ltrace` to see what calls `ls` is making. The Mac equivalent of this is `dtrace`.

Chose initially not to disable SIP

To run dtrace you will have to boot your Mac into recovery mode and run the following command so you can use `dtrace` to disable the SIP partially on the Mac to allow you to run dtrace.

 csrutil enable --without dtrace

So that the System Integrity Protection is bypassed for dtrace or if you don’t want to do that copy the ls binary and remove the code signature and then run dtrace on the copied executable.

cp /bin/ls .
sudo codesign --remove-signature ./ls-copy

`./ls-copy` exits the process without execution.

sudo ./ls-copy
                                                                                                                                                                     
[1]    8198 killed     sudo ./ls

Running `dtrace` on it gave me the following output.

sudo dtrace -c ./ls-copy 
                                                                                                                                                          
dtrace: system integrity protection is on, some features will not be available

Failed to start process notifications for pid -1 (268435459)
dtrace: failed to execute ./ls: Could not create symbolicator for task

Learnt something about Symbolicators

Symbolicators are a system for managing symbols more efficiently to dereference symbols from data which contains them (eg: raw stack traces). More on this later.

The dtrace distress continues

Preserving the PID

Even after removing the codesign we are still prevented from getting process notifcations. So, I don’t want to stop my SIP protection so, we are going to do something else. I am going to run the following script in one terminal window, find the PID (process id) of the same and run dtrace on it.

# ls-loop.sh   
                                                                                                       
#!/bin/bash

# Start a background process with the same PID
( while true; do
    ls;  # Execute ls command
    sleep 1;  # Sleep for a second to avoid rapid execution
done ) &
PID=$!  # Capture the PID of the subshell

echo "Running ls in a loop with PID: $PID"

The above will keep running the command in a loop and help us preserve the PID so that we don’t get a new PID for every run of ls.

Distrust, dtruss and dtrace

This did not work. I used dtruss and dtrace, passed the PIDs, ran the process in context and went into unnecessary tangents trying to understand the output of the dtruss command which was trying to tell me how many times the probe encountered a particular syscall which was 0.

With the above I went on a tangent for a while trying to get dtrace to work without SIP for a while and decided to

… I am a bit sleepy through so I will do it tomorrow.

(maybe some other time I will figure out how to use these)

Disabling

3nd November 2024

And now, I have disabled the SIP and the first command I am going to try is the following because that is the one which game me some output yesterday.

sudo dtruss -c ls

and now my terminal is somehow hung up. I need to restart my Mac now. I am done with this. I am going to check out dtrace and dtruss some other time. I am going to ask the Mirrord community how I can go about tracing these calls to libc a little better.

Mirrord-layer execution and hooks

Tracing hooked libc calls through mirrord’s logs

4th November 2024

It’s a new day to chase some processes and chase them I will. After getting help from the community, especially from

Aviram Hassan

for this who is the maintainer and CEO of MetalBear, the company behind Mirrord, I was able to continue with my exploration without any issues.

Trace Logging

Let’s enable trace logging for Mirrord with the following command so that we see how the internals are working in the mirrord command, and maybe we will get to see some libc function calls made by `ls`.

export RUST_LOG=mirrord=trace

where Mirrord is the target and trace is the level of logging.

I have a minikube cluster already running and upon running the command below I get the trace logs for the command.

mirrord exec -t deployment/coredns ls -n kube-system 2> trace-logs.txt

Hooks

Upon checking the logs there are quite a few things that stand out to me.

Loading into process: ls invoked as ls.
hooked "close"
hooked "close$NOCANCEL"
mirrord_layer::hooks: found "__close_nocancel" in "libutil.dylib", hooking
mirrord_layer::hooks: hook "__close_nocancel" in "libutil.dylib" failed with err Function already replaced

The above initialisation of mirrord let’s us know that the “close” libc function call has been hooked into by mirrord-layer but the ones after fail to hook into because mirrord is trying to hook into the same cloud_nocancel function call gain which are provided by the other dynamically linked libraries. Then there are more “hooked” log traces like the one below after which we get a log line saying that mirrord-layer has been initialised.

...
hooked "pwrite"
hooked "access"
hooked "faccessat"
hooked "fsync"
hooked "fdatasync"
hooked "realpath"
hooked "realpath$DARWIN_EXTSN"
hooked "lstat$INODE64"
hooked "fstat$INODE64"
hooked "stat$INODE64"
hooked "fstatat$INODE64"
hooked "fstatfs$INODE64"
hooked "fdopendir$INODE64"
hooked "readdir_r$INODE64"
hooked "readdir$INODE64"
hooked "opendir$INODE64"
Initializing mirrord-layer!

Now that all of these libc functions have been hooked into, the executable (ls) is loaded with contextual parameters. The below is from the DEBUG log. I have removed a bunch of entries from the log as well.

Environment Variables

Loaded into executable executable=Some("/var/folders/v7/yqyq_d6x2996hfnnwvgs5f080000gn/T/mirrord-bin-ghu3278mz/bin/ls") args=Some(ExecuteArgs { exec_name: "ls", invoked_as: "ls", args: ["ls"] }) pid=53099 parent_pid=1950 env_vars=Vars { inner: [("P9K_SSH", "0"), ("_P9K_TTY", "/dev/ttys010"), ("MIRRORD_PROGRESS_MODE", "off"), ("KUBE_DNS_PORT_9153_TCP_PROTO", "tcp"), ("USE_GKE_GCLOUD_AUTH_PLUGIN", "True"), ("KUBERNETES_PORT", "tcp://10.96.0.1:443"), ("PATH", "/Users/vibhav
bobade/miniconda3/bin:/Users/vibhavbobade/miniconda3/condabin:/Users/vibhavbobade/.bun/bin:..., ("KUBE_DNS_SERVICE_PORT_DNS_TCP", "53"), ("KUBE_DNS_SERVICE_PO
RT_DNS", "53"), ("TMUX_PANE", "%10"), ("ZSH", "/Users/vibhavbobade/.oh-my-zsh"), ("KUBE_DNS_PORT_53_UDP_PROTO", "udp"), ("rvmsudo_secure_path", "0"), ("SSH_AUTH_SOCK", "/private/t
mp/com.apple.launchd.12345678/Listeners"), ("TERM_PROGRAM_VERSION", "3.5a"), ("LS_COLORS", "di=1;36:ln=35:so=32:pi=33:ex=31:bd=34;46:cd=34;43:su=30;41:sg=30;46:tw=30;42:ow=30;43
"), ("XPC_FLAGS", "0x0"), ("KUBE_DNS_SERVICE_PORT_METRICS", "9153"), ("MIRRORD_CONNECT_TCP", "127.0.0.1:53938"), ("HOSTNAME", "coredns-5d78c9869d-7dp2c"), ("P9K_TTY", "old"), ("KU
BERNETES_SERVICE_PORT", "443"), ("LC_ALL", "en_US.UTF-8"), ("GOPATH", "/Users/vibhavbobade/go"), ... 
}

The above are the list of environment variables of the local process which are being passed to the mirrored process. The environment variables which stand out to me among these are the following (even if they are not important to mirrord right now, it’s helpful to understand what parts of the context are also shared by mirrord).

• ("DYLD_INSERT_LIBRARIES", "/tmp/12749644993650801669-libmirrord_layer.dylib")

  • Equivalent to LD_PRELOAD

• ("XPC_FLAGS", "0x0")

  • Used for interprocess communication in Macs.

• ("SSH_AUTH_SOCK", "/private/tmp/com.apple.launchd.12345678/Listeners")

  • Socket used by SSH agent

• ("SECURITYSESSIONID", "123456")

  • Useful for tracking user authentication and session states.

Open function calls

After the above environment variables are loaded into the mirrored remote process we can see how mirrord-layer executes the libc call hooks. We can only see the calls which are hooked in this case.

...
ThreadId(02) mirrord_layer::file::hooks: path Success(
    "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/LC_MESSAGES",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}
ThreadId(02) mirrord_layer::file::hooks: path Success(
  ".",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}
...

In the above are two of the few libc calls intercepted while running `ls`. We can tell that the first one is related reading the locale and the second one is related to reading the local `.` directory. Ok, now I have some idea of how this works (only some idea).

Mirrord-layer in this case has a replace macro in this case which is used to replace or hook the libc functions in question.

More mirrord-layer + LD_PRELOAD magic

I am not yet done staring at the logs for mirrord-layer execution. In the next one I want to try running some other command with mirrord in targeted mode to see how these different Linux Capabilities and other process features come together. Maybe we can try some file reading from remote and writing locally for that. Looking forward to seeing you in the next one.

(this discussion is continued in the next batch of relevant diary entries/post …)