Introduction to Lsof

Under Linux, many objects are considered a file, regardless of whether the object is actually a file, device, directory, or socket. Listing a file is easy, there is the shell built-in ls for that. But what if a user wanted to see which files are currently opened by the web server process? Or if that user wanted to find out which files are opened in a certain directory? That’s where lsof comes into play. Imagine lsof as a ls with the addition of “open files”.

Please note that while the BSD’s have a different utility for this job, fstat, several other flavors of Unix (Solaris, for example) also possess lsof. The options and flags are different on the other platforms, as well as the look of the output, but generally the knowledge in this article should be applicable for them too.

First, let’s take a look at the format of lsof output and how it is to be read. The usual output of lsof without any parameters would resemble the following. This has been trimmed for readability.

COMMAND    PID  TID       USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
init         1            root  cwd       DIR              254,1      4096          2 /
init         1            root  rtd       DIR              254,1      4096          2 /
init         1            root  txt       REG              254,1     36992    7077928 /sbin/init
init         1            root  mem       REG              254,1     14768    7340043 /lib/x86_64-linux-gnu/libdl-2.13.so
init         1            root  mem       REG              254,1   1603600    7340040 /lib/x86_64-linux-gnu/libc-2.13.so
init         1            root  mem       REG              254,1    126232    7340078 /lib/x86_64-linux-gnu/libselinux.so.1
init         1            root  mem       REG              254,1    261184    7340083 /lib/x86_64-linux-gnu/libsepol.so.1
init         1            root  mem       REG              254,1    136936    7340037 /lib/x86_64-linux-gnu/ld-2.13.so
init         1            root   10u     FIFO               0,14       0t0       4781 /run/initctl

These columns mean the following:

• COMMAND – The process that an open file belongs to, in this example everything is related to init.
• PID – The process identification number of said process.
• USER – The user that the process runs under. For init, it’s almost always root.
• FD – The file descriptor of the file, the most common being:
  • cwd – The current working directory (you might notice the similarity to the pwd command which prints the current working directory).
  • rtd – The root directory of a process.
  • txt – A text file, this can either mean a configuration file related to the process or the “source code” related to (or belonging to) the process.
  • mem – A so called “memory mapped file”, that means a segment of virtual memory (read: RAM) that has been assigned to a file.
  • A number – the number represents the actual file descriptor, the character after the number is the mode in which the file is opened:
  • r – Read.
  • w – Write.
  • u – Read and write.
• TYPE – Specifies the actual type of the file, the most common are:
  • REG – A regular file.
  • DIR – A directory.
  • FIFO – First in, first out.
• DEVICE – The major and minor number of the device that holds the file.
• SIZE – The size of the file, in bytes.
• NODE – The inode number of the file.
• NAME – The name of the file.

This might be a little bit overwhelming for now, but if you work with lsof a few times, it will quickly sink into your brain.

As mentioned above, the output of lsof has been shortened here. Without any arguments or filters, lsof produces hundreds of lines of output which will only leave you confused.

There are two basic approaches to solve that problem:

• Use one or more of the lsof command line options to narrow down the results.
• Pipe the output through, for example, grep.

While the latter option may sound more comfortable since you won’t have to memorize the lsof command line options, it’s generally not as flexible and efficient, so we’ll stick to the first one.

Let’s imagine that you want to open a file with your favorite text editor, and that the text editor tells you that it can only be opened in read-only mode because another program is already accessing it. lsof will help you find out who the perpetrator is:

lsof /path/to/your/file

This will produce an output similar to this:

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
vim 2679 root    5w   REG  254,1   121525 6035622 /root/lsof.txt

Apparently, you forgot to close and older session! A very similar problem happens when you try to unmount an NFS share and umount tells you it can’t because something is still accessing the mounted folder. Again, lsof can help you with identifying the culprit:

lsof +D /path/to/your/directory/

Notice the trailing slash, that’s important. Otherwise lsof will assume you mean a regular file. Don’t be confused by the + in front of the flag – lsof has so many command line options that it needs + in addition to the more common -. The output would look like this:

COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
mocp    5637  music    4r   REG   0,19 10147719 102367344 /home/Music/RMS_GNU_SONG.ogg

That means that the process mocp, with the PID 5637, belonging to the user music has opened a file called RMS_GNU_SONG.ogg. However, even after closing that process, there is still a problem – the NFS volume can’t be unmounted.

lsof has a -c flag that displays files opened an arbitrary process name.

lsof -c mocp

That would produce an output looking like this:

mocp    9383  music    4r   REG   0,19 10147719 102367344 /home/Music/ANOTHER_RMS_GNU_SONG.ogg

In this example, there is another instance of mocp running, preventing you from unmounting the share. After shutting down that process, you want to make sure that the user music has no other potentially problematic files open. lsof has a -u flag for showing files opened by a specific user. Remember, a file isn’t always just a regular file on your hard disk!

lsof -u music

You can also pass several users, separated by commas:

lsof -u music,moremusic

An important note on the default behavior of lsof: the results are OR-based, which means that you will see file results opened by processes that are owned by either the user music, or the user moremusic. If you wanted to see results matching processes that are owned by both users, then you would have to pass the flag -a:

lsof -au music, moremusic

Since both of the users are in the group musicusers, then you can also list files based on group:

lsof -g musicusers

You can also combine command line flags:

lsof -u music,moremusic -c mocp

or

lsof -u ^music +D /home/Music

In the last line, we added another special flag – ^, which stands for a logical NOT. If the output is empty after running that command, then the unmounting will most likely be successful.

In the previous examples, we mostly looked at regular files. How about sockets and network connections?

To list all current network connections lsof has the -i flag:

lsof -i

The output looks similar to what we’ve seen so far…

COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
owncloud  3509  myuser   25u  IPv4  44946      0t0  TCP strix.local:34217->myserver.vultr.com:https (ESTABLISHED)
firefox   3612  myuser   82u  IPv4  49663      0t0  TCP strix.local:43897->we-in-f100.1e100.net:https (ESTABLISHED)
ssh       3784  myuser    3u  IPv4  10437      0t0  TCP strix.local:51416->someserver.in:ssh (ESTABLISHED)
wget      4140  myuser    3w  IPv4  45586      0t0  TCP strix.local:54460->media.ccc.de:http (CLOSE_WAIT)

… except for one difference: instead of file names or directories, the column NAME now shows connection information. Each connection consists of the following parts:

• Protocol.
• Local hostname.
• Source port of the connection.
• Destination DNS name.
• Destination port.
• Status of the connection.

As with many other tools, you may opt-out of resolving DNS names and ports (-n and -P, respectively). The flag -i takes additional parameters. You can specify whether or not to show tcp, udp or icmp connections or certain ports:

lsof -i :25
or
lsof -i :smtp

Again, parameters can be combined. The following example…

lsof -i tcp:80

… will only show you TCP connections using port 80. You may also combine it with the options that you already know from “classic” files:

lsof -a -u httpd -i tcp

This will show you all TCP connections opened by the user httpd. Note the -a flag, which changes the default behavior of lsof (as mentioned earlier). As with most command-line tools, you can go extremely deep. The following will only show you TCP connections whose state is “ESTABLISHED”:

lsof -i -s TCP:ESTABLISHED

At this point, you should have a basic understanding on how lsof works, along with some common use cases. For further reading, see the manpage of lsof on your system.