tail: inotify resources exhausted

If you happen across this message while tailing a logfile:

tail: inotify resources exhausted

… And you have CrashPlan installed, then you probably have too low a limit on the number of inotify.max_user_watches. I only mention CrashPlan because this seems to be fairly common with CrashPlan on Linux. This could happen for a variety of reasons actually so to find out what is causing it, do the following:

echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
echo 1 > /sys/kernel/debug/tracing/tracing_enabled

Those two commands will enable you to “watch” inotify_add_watch events. To actually watch them, wait a few minutes after enabling, and then:

cat /sys/kernel/debug/tracing/trace

You should see some output similar to this:

root@localhost:~# cat /sys/kernel/debug/tracing/trace | more
# tracer: nop
#              | |       |          |         |
            java-13752 [010] 180569.026114: sys_inotify_add_watch -> 0x1
            java-13752 [010] 180569.038573: sys_inotify_add_watch -> 0x2
            java-13752 [010] 180569.039368: sys_inotify_add_watch -> 0x3
            java-13752 [010] 180569.044214: sys_inotify_add_watch -> 0x4
            java-13752 [010] 180569.051454: sys_inotify_add_watch -> 0x5
            java-13752 [010] 180569.052107: sys_inotify_add_watch -> 0x6
            java-13752 [010] 180569.059542: sys_inotify_add_watch -> 0x7
            java-13752 [010] 180569.060265: sys_inotify_add_watch -> 0x8
            java-13752 [010] 180569.060760: sys_inotify_add_watch -> 0x9
            java-13752 [010] 180569.068002: sys_inotify_add_watch -> 0xa
            java-13752 [010] 180569.068549: sys_inotify_add_watch -> 0xb
            java-13752 [010] 180569.082694: sys_inotify_add_watch -> 0xc
            java-13752 [010] 180569.089735: sys_inotify_add_watch -> 0xd
            java-13752 [010] 180569.093624: sys_inotify_add_watch -> 0xe
            java-13752 [010] 180569.094271: sys_inotify_add_watch -> 0xf
            java-13752 [010] 180569.098156: sys_inotify_add_watch -> 0x10
            java-13752 [010] 180569.098794: sys_inotify_add_watch -> 0x11
            java-13752 [010] 180569.105731: sys_inotify_add_watch -> 0x12
            java-13752 [010] 180569.109630: sys_inotify_add_watch -> 0x13
            java-13752 [010] 180569.119702: sys_inotify_add_watch -> 0x14
            java-13752 [010] 180569.123390: sys_inotify_add_watch -> 0x15
            java-13752 [010] 180569.127319: sys_inotify_add_watch -> 0x16
            java-13752 [010] 180569.127801: sys_inotify_add_watch -> 0x17
            java-13752 [010] 180569.131432: sys_inotify_add_watch -> 0x18
            java-13752 [010] 180569.135184: sys_inotify_add_watch -> 0x19
            java-13752 [010] 180569.135616: sys_inotify_add_watch -> 0x1a
            java-13752 [010] 180569.139202: sys_inotify_add_watch -> 0x1b
            java-13752 [010] 180569.139622: sys_inotify_add_watch -> 0x1c
            java-13752 [010] 180569.149321: sys_inotify_add_watch -> 0x1d
            java-13752 [010] 180569.149717: sys_inotify_add_watch -> 0x1e
            java-13752 [010] 180569.156260: sys_inotify_add_watch -> 0x1f
            java-13752 [010] 180569.165739: sys_inotify_add_watch -> 0x20
            java-13752 [010] 180569.169937: sys_inotify_add_watch -> 0x21
            java-13752 [010] 180569.170296: sys_inotify_add_watch -> 0x22
            java-13752 [010] 180569.177402: sys_inotify_add_watch -> 0x23
            java-13752 [010] 180569.183846: sys_inotify_add_watch -> 0x24
            java-13752 [010] 180569.187312: sys_inotify_add_watch -> 0x25
            java-13752 [010] 180569.187802: sys_inotify_add_watch -> 0x26
            java-13752 [010] 180569.191314: sys_inotify_add_watch -> 0x27
            java-13752 [010] 180569.191781: sys_inotify_add_watch -> 0x28
            java-13752 [010] 180569.198126: sys_inotify_add_watch -> 0x29
            java-13752 [010] 180569.201667: sys_inotify_add_watch -> 0x2a
            java-13752 [010] 180569.209703: sys_inotify_add_watch -> 0x2b
            java-13752 [010] 180569.212063: sys_inotify_add_watch -> 0x2c
            java-13752 [010] 180569.214432: sys_inotify_add_watch -> 0x2d
            java-13752 [010] 180569.214729: sys_inotify_add_watch -> 0x2e
            java-13752 [010] 180569.216971: sys_inotify_add_watch -> 0x2f
            java-13752 [010] 180569.219159: sys_inotify_add_watch -> 0x30
            java-13752 [010] 180569.219450: sys_inotify_add_watch -> 0x31
            java-13752 [010] 180569.221780: sys_inotify_add_watch -> 0x32
            java-13752 [010] 180569.222029: sys_inotify_add_watch -> 0x33
            java-13752 [010] 180569.225990: sys_inotify_add_watch -> 0x34
            java-13752 [010] 180569.228548: sys_inotify_add_watch -> 0x35
            java-13752 [010] 180569.228797: sys_inotify_add_watch -> 0x36
            java-13752 [010] 180569.232822: sys_inotify_add_watch -> 0x37
            java-13752 [010] 180569.233054: sys_inotify_add_watch -> 0x38
            java-13752 [010] 180569.237234: sys_inotify_add_watch -> 0x39
            java-13752 [010] 180569.237551: sys_inotify_add_watch -> 0x3a
            java-13752 [010] 180569.243332: sys_inotify_add_watch -> 0x3b
            java-13752 [010] 180569.245901: sys_inotify_add_watch -> 0x3c
            java-13752 [010] 180569.246179: sys_inotify_add_watch -> 0x3d
            java-13752 [010] 180569.250486: sys_inotify_add_watch -> 0x3e
            java-13752 [010] 180569.250802: sys_inotify_add_watch -> 0x3f
            java-13752 [010] 180569.252945: sys_inotify_add_watch -> 0x40
            java-13752 [010] 180569.253189: sys_inotify_add_watch -> 0x41
            java-13752 [010] 180569.255402: sys_inotify_add_watch -> 0x42
            java-13752 [010] 180569.255661: sys_inotify_add_watch -> 0x43
            java-13752 [010] 180569.259566: sys_inotify_add_watch -> 0x44
            java-13752 [010] 180569.261640: sys_inotify_add_watch -> 0x45
            java-13752 [010] 180569.263669: sys_inotify_add_watch -> 0x46
            java-13752 [010] 180569.265819: sys_inotify_add_watch -> 0x47
            java-13752 [010] 180569.267893: sys_inotify_add_watch -> 0x48
            java-13752 [010] 180569.269967: sys_inotify_add_watch -> 0x49
            java-13752 [010] 180569.271976: sys_inotify_add_watch -> 0x4a
            java-13752 [010] 180569.272240: sys_inotify_add_watch -> 0x4b
            java-13752 [010] 180569.291990: sys_inotify_add_watch -> 0x4c
            java-13752 [010] 180569.292369: sys_inotify_add_watch -> 0x4d
            java-13752 [010] 180569.292726: sys_inotify_add_watch -> 0x4e
            java-13752 [010] 180569.293091: sys_inotify_add_watch -> 0x4f
            java-13752 [010] 180569.293420: sys_inotify_add_watch -> 0x50
            java-13752 [010] 180569.293749: sys_inotify_add_watch -> 0x51
            java-13752 [010] 180569.305760: sys_inotify_add_watch -> 0x52
            java-13752 [010] 180569.306204: sys_inotify_add_watch -> 0x53
            java-13752 [010] 180569.306665: sys_inotify_add_watch -> 0x54
            java-13752 [010] 180569.307042: sys_inotify_add_watch -> 0x55
            java-13752 [010] 180569.307385: sys_inotify_add_watch -> 0x56
            java-13752 [010] 180569.307724: sys_inotify_add_watch -> 0x57
            java-13752 [010] 180569.308032: sys_inotify_add_watch -> 0x58
            java-13752 [010] 180569.321561: sys_inotify_add_watch -> 0x59
            java-13752 [010] 180569.321968: sys_inotify_add_watch -> 0x5a
            java-13752 [010] 180569.322274: sys_inotify_add_watch -> 0x5b
            java-13752 [010] 180569.322552: sys_inotify_add_watch -> 0x5c
            java-13752 [010] 180569.322830: sys_inotify_add_watch -> 0x5d
            java-13752 [010] 180569.323106: sys_inotify_add_watch -> 0x5e
            java-13752 [010] 180569.323378: sys_inotify_add_watch -> 0x5f
            java-13752 [010] 180569.323635: sys_inotify_add_watch -> 0x60
            java-13752 [010] 180569.337109: sys_inotify_add_watch -> 0x61
            java-13752 [010] 180569.337452: sys_inotify_add_watch -> 0x62
            java-13752 [010] 180569.337779: sys_inotify_add_watch -> 0x63
            java-13752 [010] 180569.338094: sys_inotify_add_watch -> 0x64
            java-13752 [010] 180569.338379: sys_inotify_add_watch -> 0x65
            java-13752 [010] 180569.338660: sys_inotify_add_watch -> 0x66

Note the task and PID columns:

root@localhost:~# ps waux | grep java
root     13679 50.3  4.1 6393844 510320 pts/1  SNl  11:58   1:18 /usr/local/crashplan/jre/bin/java -Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx1024m -Djava.net.preferIPv4Stack=true -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0 -Dc42.native.md5.enabled=false -classpath /usr/local/crashplan/lib/com.backup42.desktop.jar:/usr/local/crashplan/lang com.backup42.service.CPService

The PID doesn’t always match up with the process that added the watch, in the example above, CrashPlan likely spawned a child process (PID: 13752, according to our trace) to add the inotify watches.

So now you know why this is happening, here is what you should do about it, First, to see what the currently configured limit is:

cat /proc/sys/fs/inotify/max_user_watches

It seems that the default limit for Ubuntu servers is 8192. To raise the limit, run the following as root:

sysctl -w fs.inotify.max_user_watches=32768

Or, to make the limit permanent, edit /etc/sysctl.conf and append the following line:


The limit 32768 is a bit high so you may want a lower one depending on the available resources (RAM, CPU, etc.) of your machine. For reference, I use this configuration on production servers with 12GB RAM or more. YMMV.

To put things back to their default settings (defaults for Ubuntu anyway):

echo 0 > /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
echo 1 > /sys/kernel/debug/tracing/tracing_enabled

(On ubuntu, the default setting for /sys/kernel/debug/tracing/tracing_enabled is “1″)

Zimbra weirdness when you zmcontrol start/stop from the wrong directory

Aug 21 15:48:01 mail zmconfigd[50891]: Exception in bin/zmmtactl: (Cannot run program "/opt/zimbra/bin/zmmtactl" (in directory "/root"): error=13, Permission denied)
Aug 21 15:48:01 mail zmconfigd[50891]: Exception in bin/zmopendkimctl: (Cannot run program "/opt/zimbra/bin/zmopendkimctl" (in directory "/root"): error=13, Permission denied)
Aug 21 15:48:01 mail zmconfigd[50891]: Exception in bin/zmsaslauthdctl: (Cannot run program "/opt/zimbra/bin/zmsaslauthdctl" (in directory "/root"): error=13, Permission denied)
Aug 21 15:48:01 mail zmconfigd[50891]: Exception in bin/zmswatchctl: (Cannot run program "/opt/zimbra/bin/zmswatchctl" (in directory "/root"): error=13, Permission denied)
Aug 21 15:48:01 mail zmconfigd[50891]: Exception in bin/zmspellctl: (Cannot run program "/opt/zimbra/bin/zmspellctl" (in directory "/root"): error=13, Permission denied)
Aug 21 15:48:01 mail zmconfigd[50891]: Exception in bin/zmstatctl: (Cannot run program "/opt/zimbra/bin/zmstatctl" (in directory "/root"): error=13, Permission denied)
Aug 21 15:48:01 mail zmconfigd[50891]: Exception in bin/zmclamdctl: (Cannot run program "/opt/zimbra/bin/zmclamdctl" (in directory "/root"): error=13, Permission denied)

Seeing similar entries in your /var/log/zimbra.log? This happens when you do a zmcontrol start/stop/restart from a directory other than /opt/zimbra.

Anyway here’s what you do to “fix” it:

*As root:

zmcontrol stop
/opt/zimbra/libexec/zmfixperms --verbose --extended
su -l zimbra

It may not be necessary to fix permissions with zmfixperms but do it just in case. Next, make sure that you are in /opt/zimbra; if not, cd into that directory, then:

zmcontrol start

Setting password policies in Zimbra when using local password fallback

Zimbra’s web admin doesn’t let you modify the password settings for a COS if at least one domain is configured to use LDAP/Active Directory for authentication, regardless of whether zimbraAuthFallbackToLocal is TRUE for that domain. I’m sure this is a bug in the interface, but never fear, the command line will save us.

Simply login to your server, then:

su -l zimbra
zmprov gac -v | grep Password | sort | uniq

That will show you all the attributes you can set. I use sort + uniq in the above example because the ‘gac’ option to zmprov means “get all COS” and the -v means show all attributes and their values. Since I have multiple classes of service, that would display all attributes and their values for all classes of service. In the example above, we only want to know what attributes are available to configure:

zimbraFeatureChangePasswordEnabled: TRUE
zimbraMobilePolicyAllowSimpleDevicePassword: FALSE
zimbraMobilePolicyAlphanumericDevicePasswordRequired: FALSE
zimbraMobilePolicyDevicePasswordEnabled: TRUE
zimbraMobilePolicyDevicePasswordExpiration: 0
zimbraMobilePolicyDevicePasswordHistory: 8
zimbraMobilePolicyMaxDevicePasswordFailedAttempts: 4
zimbraMobilePolicyMinDevicePasswordComplexCharacters: 0
zimbraMobilePolicyMinDevicePasswordLength: 4
zimbraMobilePolicyPasswordRecoveryEnabled: TRUE
zimbraPasswordEnforceHistory: 10
zimbraPasswordLocked: FALSE
zimbraPasswordLockoutDuration: 1h
zimbraPasswordLockoutEnabled: FALSE
zimbraPasswordLockoutEnabled: TRUE
zimbraPasswordLockoutFailureLifetime: 1h
zimbraPasswordLockoutMaxFailures: 10
zimbraPasswordMaxAge: 0
zimbraPasswordMaxLength: 64
zimbraPasswordMinAge: 0
zimbraPasswordMinAlphaChars: 1
zimbraPasswordMinDigitsOrPuncs: 1
zimbraPasswordMinLength: 18
zimbraPasswordMinLowerCaseChars: 1
zimbraPasswordMinNumericChars: 1
zimbraPasswordMinPunctuationChars: 1
zimbraPasswordMinUpperCaseChars: 1

The name of the attributes pretty much sum up what they do, so let’s say you wanted to set the minimum required length of passwords for the COS named “example.com” to 18 characters, here’s the command you’d use:

zmprov mc example.com zimbraPasswordMinLength 18

For more information, type:

zmprov help cos

Multitail color scheme for Zimbra’s mailbox.log

If you’ve ever spent time tailing log files you’ve probably heard of multitail (if not, go check it out). You can install it via apt on Debian — not sure about other OSs. Anyway, here’s a color scheme for Zimbra’s mailbox.log that you can paste into /etc/multitail.conf:

# zimbra mailbox.log
colorscheme:zimbra:Zimbra mailbox.log
cs_re_s:green,,bold: (DEBUG)
cs_re_s:green,,bold: (INFO)
cs_re_s:yellow,,bold: (WARN)
cs_re_s:red,,bold: (ERROR)
cs_re_s:yellow,red,bold: (FATAL)
cs_re_s:cyan:^([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}\:[0-9]{2}\:[0-9]{2},[0-9]*?)


Active Directory users aren’t able to update their password via SonicWall NetExtender

This seems to be a problem for a lot of people on the ‘net so I figured I’d document it here.

First, we need to make sure LDAP authentication is setup properly; take a look at the following screenshots (this assumes that you’ve already created a separate account for binding to the AD/LDAP database… in my examples below, this account is referred to as the “ldap bind” account.)

Security Settings

Security Settings (from https://www.fuzeqna.com/sonicwallkb/ext/kbdetail.aspx?kbid=5097)

Make to select 'LDAP' and not 'LDAP + Local Users' .

Make to select ‘LDAP’ and not ‘LDAP + Local Users’ .

Use the "display name" of the user account that you'd like to use for binding to the Active Directory server. Make sure that this account does NOT have Administrative privileges.

Use the “display name” of the user account that you’d like to use for binding to the Active Directory server. Make sure that this account does NOT have Administrative privileges.

Click "Import user groups" and select the group that contains the users who will should be granted SSLVPN access. Then set that group as the "Default LDAP User Group". I usually create a separate group for this purpose.

Click “Import user groups” and select the group that contains the users who will should be granted SSLVPN access. Then set that group as the “Default LDAP User Group”. I usually create a separate group for this purpose.

This is where you'll configure the locations of the OUs that contain Groups and Users who will have SSLVPN access.

This is where you’ll configure the locations of the OUs that contain Groups and Users who will have SSLVPN access.

Once you’ve done all of that, you’ll need to delegate the ability to change user passwords to your “ldap bind” account. This “sandboxes” the account; i.e., it allows the account to be able to change passwords without requiring administrative privileges. This way, the account only has the exact level of privilege it needs to do its job. To do this, right-click on the parent container of the objects you want to control, then select “delegate control”. Click “Next” then click “Add” and select the “ldap bind” account, then click “Next” again. Check the box next to “Reset user passwords and force password change at next logon”, then click “Next” and “Finish”.

If for some reason it’s still not working, make sure that you have checked the “Include inheritable permissions from this object’s parent” box, from the “Advanced Security Settings” page on each user object that needs to be able to change their password from SonicWall NetExtender.

Advanced Security Settings for [user object]

If you delegated the ability to change/reset passwords to your ldap bind account on the parent container of the target user objects, then each child user object must have this option selected. Alternatively, you could add the ldap bind account to the security settings on each user, manually.

That should be all there is to it!