Recovering tmpfs from Linux and Android Memory Captures with Volatility

Tuesday, August 14, 2012

Andrew Case


In this blog post I will introduce a new Volatility Linux plugin, tmpfs, and discuss its uses and implementation.

The purpose of this plugin, which can currently be found here, is to reconstruct any tmpfs filesystem contained within a Linux memory capture and fully recover it to disk.   

What is tmpfs?

Tmpfs is a Linux filesystem whose contents reside only in memory. This means that files and directories inside of tmpfs mounts are never written to the local disk and that once a tmpfs mount is unmounted, the entire filesystem is “gone”.

Why is tmpfs interesting?

Tmpfs is interesting from a forensics perspective for a few reasons. The first is that, in a traditional forensics scenario, the investigator expects that he can shut a computer off, images its disk(s), and get back the filesystem at the time of when the computer was running. With tmpfs, this is obviously not true…  

The second reason that tmpfs is interesting is because of how it is used by Linux distributions as well as by attackers and certain malware samples.  On many Linux distributions, the /tmp directory is mounted as tmpfs so that the directory will be cleared on reboot.

This helps speedup the boot process as the init code does not need to unlink every file and directory under /tmp, but it also means that the traditional (disk only) preservation process will miss any files stored in /tmp. This directory often contains scratch data and install files for many applications as well as for rootkits/malware that try to be agnostic to the distribution they are being installed on.   

Linux also uses tmpfs to implement shared memory through /dev/shm. While recovery of this directory may help in recovering IPC data, its main purpose related to forensics & IR is that it is often used as a scratch directory by attackers to download files, compile programs, and to store the output of commands and malware hooks.  Since the filesystem goes away on reboot, attackers know they can frustrate and/or defeat traditional forensics investigations using this method.

The third reason that tmpfs is of interest is that Linux live CDs use tmpfs as the underlying store for all changes made to the filesystem after the machine boots, since obviously it cannot write back to the CD. If you are interested in reading about memory analysis of live CDs, I did a presentation and white paper on it at BlackHat DC 2011 ( slides | paper ) so I will not recap it all here.  The paper also discusses much of the kernel internals described at the end of this blog post.

Finally, while still being researched, it has been discovered that many Android applications use a tmpfs instance created by the Android runtime to store “interesting” runtime data and state.  The tmpfs plugin has already been tested on a number of phones using Volatility’s Android support and the data was verified to be fully and correctly recovered on all devices.  For the latest details on this, please contact Joe Sylve (@jtsylve).

How do you use the plugin?

The plugin requires two steps to work. First, you run it with the ‘-L’ option to list the tmpfs filesystems within the image:  

python --profile=Linuxthisx86 -f /root/lime-tmpfs linux_tmpfs –L

  The output of this invocation on my test VM looks like this:  

Volatile Systems Volatility Framework 2.1_rc3 1 -> /lib/init/rw 2 -> /dev/shm

Now, let us say I want to recover the contents of /dev/shm, so I then run the plugin with the –S option of ‘2’ and an output directory:  

python --profile=Linuxthisx86 -f /root/lime-tmpfs linux_tmpfs -S 2 -o outputdir/

When this command is finished, the entire filesystem of “/dev/shm”, including all files, directories, and metadata (atime, mtime, permissions), will be replicated to the “outputdir” directory.  In a real forensics investigation, “outputdir” should be on an external drive or a separate mount point that can be remounted read-only after the filesystem is recovered.


When recovering the filesystem from memory to a disk there are a few limitations that investigators should be aware of:

1) While the script does preserve the modified and accessed times of the recovered files and directories, it has no method in which to preserve the create times. This is because the Python os.utime function only supports modifying the accessed and modified times. In order to change the create times on a standard Linux filesystem, the script would need to interact with something along the lines of debugfs, which can be error-prone and only supports the Ext family of filesystems.  Similar burdens are faced on Windows and Mac filesystems as well.

The result of this limitation is that the create time of all entries in the filesystem will actually be the time that the Volatility plugin created them. If enough interest is shown in the plugin and this limitation, I can easily extend the plugin to support writing out a mactime file with all the correct information.

2) The owner/group information is not preserved for recovered files. This may be added to future versions of the script, at least if the output filesystem is Linux based.  

3) There is currently no support for recovering deleted files within tmpfs stores.

How does the plugin work?

Now that we have explained the purpose of the plugin, it is time to dig into some Linux kernel internal in order to explain the plugin's operation. If you are not a programmer or not interested in kernel internals, you can freely skip to the end of the post.

For the plugin to work it needs to be able to the following:

1) Find and list all tmpfs mount points in the memory image

2) Recursively enumerate and reconstruct the entire filesystem for every tmpfs instance

3) Record the needed metadata of each file and directory and update the recovered files on disk to match it

4) For every file found, recover the file contents and write them to disk


1) Finding all tmpfs mount points

The tmpfs mount points are found by leveraging the capabilities of the mount plugin (volatility/plugins/linux/ The mount plugin works by walking the mount_hashtable  hash table, which stores a vfsmount structure for every mount point. This is the same operation the kernel does at runtime to populate the /proc/mounts file.  The tmpfs plugin then filters the vfsmount structures to only those representing tmpfs mounts.  This is all that is needed to implement the ‘-L’ option of the plugin.

2) Recurse the filesystem

Each vfsmount structure contains a pointer to the root directory entry of the filesystem, of type dentry, in its mnt_root member. dentry structures are generic structures used to represent files, directories, pipes, sockets, and all other file types within the kernel. They have no 1:1 mapping on disk and there can be multiple dentry structures per inode. Once the root dentry structure is obtained, we can then recursively traverse the entire filesystem by walking each dentry’s d_u.d_child member. For each entry in the child list, we check if it’s a directory or a file. If it’s a directory, the plugin simply creates the directory in the output directory’s hierarchy and then recursively processes it. If it’s a file, then we need to recover its contents from memory and create the file in the output hierarchy.  For both directories and files, we need to update the metadata of the newly created files to match those in memory. This process effectively recovers the entire filesystem to disk in the exact order and layout as in memory.

3) Recovering metadata

Recovering metadata is a simple process because all the information is directly stored within the inode structure of the file or directory, which is stored in the d_inode member of the dentry structure.  This line of code from the plugin accesses all the metadata from the inode that is currently recovered:  

(perms, size, atime, mtime) = (inode.i_mode, inode.i_size, inode.i_atime, inode.i_mtime)

4) Recovering file contents

In order to recover a particular file’s contents, we need to be able to locate the structures that track its memory pages within the kernel’s page cache and then find the actual physical pages that store the contents.

The page cache is a performance enhancing mechanism used by the kernel to cache physical pages in RAM so that they do not have to be read from the raw device or network device on which they are stored each time they are accessed. For example, on an active SSH server, a number of files (passwd, shadow, sshd_config, /bin/bash, etc) are going to be constantly read as users log in and off the system.

If the kernel had to fully read in these files from disk to memory each time a user logged in, performance would be unacceptably bad. To alleviate this, pages that are read in from disk are then put in the page cache, so the next time an applications needs to be executed or reads a file, the kernel can simply return the page from the cache without having to touch the disk.

An aside: If you have ever dd’ed a disk image in Linux and watched ‘top’, you may have noticed that your entire RAM contents was being used inside the kernel. Much of that was the kernel reading ahead on the disk being imaged and filling the page cache with the next blocks to be processed by dd.

Since the filesystem type we want to recover (tmpfs) is stored only in memory, it is implemented so that all its pages are always stored within the page cache. This lets the filesystem work as all others do, and the kernel will never look for a disk or other source to read from because the files will always be up-to-date within the cache.

In order to locate the pages of a particular file, we need to use its index into the page cache. This is stored within the dentry.d_inode.i_mapping member which is of type address_spaceaddress_space structures are used to track sets of sparse physical pages that together form a contiguous data unit (in this case, a file within a tmpfs mount).  To access the pages of a file inside its address_space, we need to walk the radix tree of struct pages stored within the address_space’s page_tree member. A struct page tracks a physical page and lets the kernel determine the physical address of the page’s contents.

To find the correct index in the tree for a page, we simply divide the offset of the page into the file by page size (4k in this case). We do this in a loop for the entire file in order to recover each page sequentially.  I will save the reader the pain of having to understand the kernel’s radix tree, but the interested reader can look at the radix_tree_lookup_slot function in the tmpfs plugin.  The page cache and its storage format is also very nicely explained in Understanding the Linux Kernel 3rd Edition. Once we are able to walk the page_tree of a particular file and recover each index (a struct page), we then need to determine the physical address of the page it tracks.

Finding the physical address is performed by indexing the struct page into the mem_map array. This array holds a struct page for every physical page of memory. Indexing into mem_map will give us the page number of the physical page, which we can then shift left by PAGE_SHIFT (12) in order to get the page’s physical offset. This process roughly corresponds to the page_to_pfn macro within the kernel.

Once we have the physical offset, all we have to do is read it using Volatility’s API and then concatenate each page’s contents together into the file. At this point we will have recovered the entire filesystem along with its contents and metadata.  


This blog post has showcased a new and exciting Volatility plugin. Being able to fully recover tmpfs filesystems out of memory greatly adds to the depth of Volatility’s Linux support for both traditional Linux installs as well as for Android devices.  By incorporating this plugin into your forensics and incident response processes, you will recover information that until now has large been ignored or missing.

If you have any questions or comments about this post, please reply in the comments section or you catch me on Twitter (@attrc).

Cross-posted from Memory Forensics Blog

Possibly Related Articles:
Information Security
Forensics Tools Penetration Testing Linux Android Network Security Data Recovery Tutorial tmpfs
Post Rating I Like this!
Kevin McAleavey Interesting "toy" you have there. illustrates very well the reason why we decided to go with BSD as the starting point for our KNOS Secure operating system years ago. We were quite surprised when we discovered how quickly we were able to exploit the Air Force's "LPS version 1.2.5" (which is also a tmpfs-based "live-cd") through its included Java code last year. tmpfs, because it is swap-backed and mounted as an actual file system always had the risk of anyone (not only forensics people, but the bad guys as well) finnding the inodes and having their way with them.

One of the advantages to our design in KNOS is that we went with BSD's MDMFS which doesn't present external hooks and allows us to keep the OS layer away from any userland applications such as yours for probing. I gave "volatility" a shot and of course since it contains no native BSD hooks at this time, it failed to do what it's supposed to. I'd be quite curious though to see how it works against the BSD kernel when you get around to it - keeping KNOS free of intrusions is a critical part of our mission. Especially given that Linux isn't up to the task either and we like to be very certain of our own impenetrability since that's highly critical to our customers.

And before anyone misunderstand my point, we definitely intend to keep the "good guys" out of KNOS because if the "good guys" can break into an operating system, then you KNOW the "bad guys" will likely get in there first. This is critical to us here.
Andrew Case Hello,

First, I wanted to say thanks for reading the article and taking the time to comment.

While I am not a FreeBSD internals expert, I took a quick look at the tmpfs implementation within FreeBSD and it very similar to the way Linux does it. While you can defend against userspace attacks from reading the filesystem, once someone gets access to kernel memory (either through something like /dev/mem, a kernel driver, or a kernel level exploit), all bets are off and the filesystem will be recoverable.

Volatility is meant to normally be used against offline memory captures and not directly against a running machine. If it is used on a running machine it needs access to some device, such as /dev/mem, that allows for reading kernel memory.

So I think your biggest concerns will be:

1) Making sure that people cannot load drivers after the system boots (this is similar to what grsecurity does). This includes not letting root users load drivers at all. If they can, then the raw filesystem in memory can be accessed.

2) Making sure that /dev/mem, /dev/kmem, and any similar devices on FreeBSD are not turned on.

You also have to consider physical security because if I can walk up to your machine and access a firewire port then I can get a memory capture regardless of security protections done by the OS. At that point, plugins could be added to Volatility that would be able to locate and pull the filesystem out of memory since it would have access to the kernel's address space.
Kevin McAleavey And thanks for the reply!

The old saying, "once someone can physically access your machine, it's no longer yours" remains true as always. :)

While BSD does provide "kmem," in our design there is no root user and kmem is not provided. But the salient point is that MDMFS is not the same as TMPFS and is an entirely different construct. In KNOS, we also do not provide shared libraries or memory which defeats the ability to hook or access memory which does not belong to the specific process. Kernel debug code and hooks also do not exist in our design. That's why what you're doing fascinates me the way it does.

Please do let us know if you add BSD code, I'd certainly like to give it a spin just to see if there's anything else we need to seal up.
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.