tail: “inotify resources exhausted” and/or “inotify cannot be used, reverting to polling: Too many open files”

If you happen across either of these messages while tailing a logfile:

  • tail: inotify resources exhausted
  • tail: inotify cannot be used, reverting to polling: Too many open files

… And you have CrashPlan installed[*], then you probably have too low a limit on the number of inotify.max_user_watches. I only mention CrashPlan because this seems to be fairly common with CrashPlan on Linux. This could happen for a variety of reasons actually so to find out what is causing it, do the following:

echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
echo 1 > /sys/kernel/debug/tracing/tracing_enabled

Those two commands will enable you to “watch” inotify_add_watch events. To actually watch them, wait a few minutes after enabling, and then:

cat /sys/kernel/debug/tracing/trace

You should see some output similar to this:

root@localhost:~# cat /sys/kernel/debug/tracing/trace | more
# tracer: nop
#
#           TASK-PID    CPU#    TIMESTAMP  FUNCTION
#              | |       |          |         |
            java-13752 [010] 180569.026114: sys_inotify_add_watch -> 0x1
            java-13752 [010] 180569.038573: sys_inotify_add_watch -> 0x2
            java-13752 [010] 180569.039368: sys_inotify_add_watch -> 0x3
            java-13752 [010] 180569.044214: sys_inotify_add_watch -> 0x4
            java-13752 [010] 180569.051454: sys_inotify_add_watch -> 0x5
            java-13752 [010] 180569.052107: sys_inotify_add_watch -> 0x6
            java-13752 [010] 180569.059542: sys_inotify_add_watch -> 0x7
            java-13752 [010] 180569.060265: sys_inotify_add_watch -> 0x8
            java-13752 [010] 180569.060760: sys_inotify_add_watch -> 0x9
            java-13752 [010] 180569.068002: sys_inotify_add_watch -> 0xa
            java-13752 [010] 180569.068549: sys_inotify_add_watch -> 0xb
            java-13752 [010] 180569.082694: sys_inotify_add_watch -> 0xc
            java-13752 [010] 180569.089735: sys_inotify_add_watch -> 0xd
            java-13752 [010] 180569.093624: sys_inotify_add_watch -> 0xe
            java-13752 [010] 180569.094271: sys_inotify_add_watch -> 0xf
            java-13752 [010] 180569.098156: sys_inotify_add_watch -> 0x10
            java-13752 [010] 180569.098794: sys_inotify_add_watch -> 0x11
            java-13752 [010] 180569.105731: sys_inotify_add_watch -> 0x12
            java-13752 [010] 180569.109630: sys_inotify_add_watch -> 0x13
            java-13752 [010] 180569.119702: sys_inotify_add_watch -> 0x14
            java-13752 [010] 180569.123390: sys_inotify_add_watch -> 0x15
            java-13752 [010] 180569.127319: sys_inotify_add_watch -> 0x16
            java-13752 [010] 180569.127801: sys_inotify_add_watch -> 0x17
            java-13752 [010] 180569.131432: sys_inotify_add_watch -> 0x18
            java-13752 [010] 180569.135184: sys_inotify_add_watch -> 0x19
            java-13752 [010] 180569.135616: sys_inotify_add_watch -> 0x1a
            java-13752 [010] 180569.139202: sys_inotify_add_watch -> 0x1b
            java-13752 [010] 180569.139622: sys_inotify_add_watch -> 0x1c
            java-13752 [010] 180569.149321: sys_inotify_add_watch -> 0x1d
            java-13752 [010] 180569.149717: sys_inotify_add_watch -> 0x1e
            java-13752 [010] 180569.156260: sys_inotify_add_watch -> 0x1f
            java-13752 [010] 180569.165739: sys_inotify_add_watch -> 0x20
            java-13752 [010] 180569.169937: sys_inotify_add_watch -> 0x21
            java-13752 [010] 180569.170296: sys_inotify_add_watch -> 0x22
            java-13752 [010] 180569.177402: sys_inotify_add_watch -> 0x23
            java-13752 [010] 180569.183846: sys_inotify_add_watch -> 0x24
            java-13752 [010] 180569.187312: sys_inotify_add_watch -> 0x25
            java-13752 [010] 180569.187802: sys_inotify_add_watch -> 0x26
            java-13752 [010] 180569.191314: sys_inotify_add_watch -> 0x27
            java-13752 [010] 180569.191781: sys_inotify_add_watch -> 0x28
            java-13752 [010] 180569.198126: sys_inotify_add_watch -> 0x29
            java-13752 [010] 180569.201667: sys_inotify_add_watch -> 0x2a
            java-13752 [010] 180569.209703: sys_inotify_add_watch -> 0x2b
            java-13752 [010] 180569.212063: sys_inotify_add_watch -> 0x2c
            java-13752 [010] 180569.214432: sys_inotify_add_watch -> 0x2d
            java-13752 [010] 180569.214729: sys_inotify_add_watch -> 0x2e
            java-13752 [010] 180569.216971: sys_inotify_add_watch -> 0x2f
            java-13752 [010] 180569.219159: sys_inotify_add_watch -> 0x30
            java-13752 [010] 180569.219450: sys_inotify_add_watch -> 0x31
            java-13752 [010] 180569.221780: sys_inotify_add_watch -> 0x32
            java-13752 [010] 180569.222029: sys_inotify_add_watch -> 0x33
            java-13752 [010] 180569.225990: sys_inotify_add_watch -> 0x34
            java-13752 [010] 180569.228548: sys_inotify_add_watch -> 0x35
            java-13752 [010] 180569.228797: sys_inotify_add_watch -> 0x36
            java-13752 [010] 180569.232822: sys_inotify_add_watch -> 0x37
            java-13752 [010] 180569.233054: sys_inotify_add_watch -> 0x38
            java-13752 [010] 180569.237234: sys_inotify_add_watch -> 0x39
            java-13752 [010] 180569.237551: sys_inotify_add_watch -> 0x3a
            java-13752 [010] 180569.243332: sys_inotify_add_watch -> 0x3b
            java-13752 [010] 180569.245901: sys_inotify_add_watch -> 0x3c
            java-13752 [010] 180569.246179: sys_inotify_add_watch -> 0x3d
            java-13752 [010] 180569.250486: sys_inotify_add_watch -> 0x3e
            java-13752 [010] 180569.250802: sys_inotify_add_watch -> 0x3f
            java-13752 [010] 180569.252945: sys_inotify_add_watch -> 0x40
            java-13752 [010] 180569.253189: sys_inotify_add_watch -> 0x41
            java-13752 [010] 180569.255402: sys_inotify_add_watch -> 0x42
            java-13752 [010] 180569.255661: sys_inotify_add_watch -> 0x43
            java-13752 [010] 180569.259566: sys_inotify_add_watch -> 0x44
            java-13752 [010] 180569.261640: sys_inotify_add_watch -> 0x45
            java-13752 [010] 180569.263669: sys_inotify_add_watch -> 0x46
            java-13752 [010] 180569.265819: sys_inotify_add_watch -> 0x47
            java-13752 [010] 180569.267893: sys_inotify_add_watch -> 0x48
            java-13752 [010] 180569.269967: sys_inotify_add_watch -> 0x49
            java-13752 [010] 180569.271976: sys_inotify_add_watch -> 0x4a
            java-13752 [010] 180569.272240: sys_inotify_add_watch -> 0x4b
            java-13752 [010] 180569.291990: sys_inotify_add_watch -> 0x4c
            java-13752 [010] 180569.292369: sys_inotify_add_watch -> 0x4d
            java-13752 [010] 180569.292726: sys_inotify_add_watch -> 0x4e
            java-13752 [010] 180569.293091: sys_inotify_add_watch -> 0x4f
            java-13752 [010] 180569.293420: sys_inotify_add_watch -> 0x50
            java-13752 [010] 180569.293749: sys_inotify_add_watch -> 0x51
            java-13752 [010] 180569.305760: sys_inotify_add_watch -> 0x52
            java-13752 [010] 180569.306204: sys_inotify_add_watch -> 0x53
            java-13752 [010] 180569.306665: sys_inotify_add_watch -> 0x54
            java-13752 [010] 180569.307042: sys_inotify_add_watch -> 0x55
            java-13752 [010] 180569.307385: sys_inotify_add_watch -> 0x56
            java-13752 [010] 180569.307724: sys_inotify_add_watch -> 0x57
            java-13752 [010] 180569.308032: sys_inotify_add_watch -> 0x58
            java-13752 [010] 180569.321561: sys_inotify_add_watch -> 0x59
            java-13752 [010] 180569.321968: sys_inotify_add_watch -> 0x5a
            java-13752 [010] 180569.322274: sys_inotify_add_watch -> 0x5b
            java-13752 [010] 180569.322552: sys_inotify_add_watch -> 0x5c
            java-13752 [010] 180569.322830: sys_inotify_add_watch -> 0x5d
            java-13752 [010] 180569.323106: sys_inotify_add_watch -> 0x5e
            java-13752 [010] 180569.323378: sys_inotify_add_watch -> 0x5f
            java-13752 [010] 180569.323635: sys_inotify_add_watch -> 0x60
            java-13752 [010] 180569.337109: sys_inotify_add_watch -> 0x61
            java-13752 [010] 180569.337452: sys_inotify_add_watch -> 0x62
            java-13752 [010] 180569.337779: sys_inotify_add_watch -> 0x63
            java-13752 [010] 180569.338094: sys_inotify_add_watch -> 0x64
            java-13752 [010] 180569.338379: sys_inotify_add_watch -> 0x65
            java-13752 [010] 180569.338660: sys_inotify_add_watch -> 0x66
--More--

Note the task and PID columns:

root@localhost:~# ps waux | grep java
root     13679 50.3  4.1 6393844 510320 pts/1  SNl  11:58   1:18 /usr/local/crashplan/jre/bin/java -Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx1024m -Djava.net.preferIPv4Stack=true -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0 -Dc42.native.md5.enabled=false -classpath /usr/local/crashplan/lib/com.backup42.desktop.jar:/usr/local/crashplan/lang com.backup42.service.CPService

The PID doesn’t always match up with the process that added the watch, in the example above, CrashPlan likely spawned a child process (PID: 13752, according to our trace) to add the inotify watches.

So now you know why this is happening, here is what you should do about it, First, to see what the currently configured limit is:

cat /proc/sys/fs/inotify/max_user_watches

It seems that the default limit for Ubuntu servers is 8192. To raise the limit, run the following as root:

sysctl -w fs.inotify.max_user_watches=32768

Or, to make the limit permanent, edit /etc/sysctl.conf and append the following line:

fs.inotify.max_user_watches=32768

Then be sure to re-load the config file using the following command:

sysctl -p

The limit 32768 might be a bit high[**] so you may want a lower one depending on the available resources (RAM, CPU, etc.) of your machine. For reference, I use this configuration on production servers with 12GB RAM or more. YMMV.

To put things back to their default settings (defaults for Ubuntu anyway):

echo 0 > /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
echo 1 > /sys/kernel/debug/tracing/tracing_enabled

(On ubuntu, the default setting for /sys/kernel/debug/tracing/tracing_enabled is “1”)

Notes / Further reading: