tail: “inotify resources exhausted” and/or “inotify cannot be used, reverting to polling: Too many open files”

If you happen across either of these messages while tailing a logfile:

  • tail: inotify resources exhausted
  • tail: inotify cannot be used, reverting to polling: Too many open files

… And you have CrashPlan installed[*], then you probably have too low a limit on the number of inotify.max_user_watches. I only mention CrashPlan because this seems to be fairly common with CrashPlan on Linux. This could happen for a variety of reasons actually so to find out what is causing it, do the following:

echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
echo 1 > /sys/kernel/debug/tracing/tracing_enabled

Those two commands will enable you to “watch” inotify_add_watch events. To actually watch them, wait a few minutes after enabling, and then:

cat /sys/kernel/debug/tracing/trace

You should see some output similar to this:

[email protected]:~# cat /sys/kernel/debug/tracing/trace | more
# tracer: nop
#              | |       |          |         |
            java-13752 [010] 180569.026114: sys_inotify_add_watch -> 0x1
            java-13752 [010] 180569.038573: sys_inotify_add_watch -> 0x2
            java-13752 [010] 180569.039368: sys_inotify_add_watch -> 0x3
            java-13752 [010] 180569.044214: sys_inotify_add_watch -> 0x4
            java-13752 [010] 180569.051454: sys_inotify_add_watch -> 0x5
            java-13752 [010] 180569.052107: sys_inotify_add_watch -> 0x6
            java-13752 [010] 180569.059542: sys_inotify_add_watch -> 0x7
            java-13752 [010] 180569.060265: sys_inotify_add_watch -> 0x8
            java-13752 [010] 180569.060760: sys_inotify_add_watch -> 0x9
            java-13752 [010] 180569.068002: sys_inotify_add_watch -> 0xa
            java-13752 [010] 180569.068549: sys_inotify_add_watch -> 0xb
            java-13752 [010] 180569.082694: sys_inotify_add_watch -> 0xc
            java-13752 [010] 180569.089735: sys_inotify_add_watch -> 0xd
            java-13752 [010] 180569.093624: sys_inotify_add_watch -> 0xe
            java-13752 [010] 180569.094271: sys_inotify_add_watch -> 0xf
            java-13752 [010] 180569.098156: sys_inotify_add_watch -> 0x10
            java-13752 [010] 180569.098794: sys_inotify_add_watch -> 0x11
            java-13752 [010] 180569.105731: sys_inotify_add_watch -> 0x12
            java-13752 [010] 180569.109630: sys_inotify_add_watch -> 0x13
            java-13752 [010] 180569.119702: sys_inotify_add_watch -> 0x14
            java-13752 [010] 180569.123390: sys_inotify_add_watch -> 0x15
            java-13752 [010] 180569.127319: sys_inotify_add_watch -> 0x16
            java-13752 [010] 180569.127801: sys_inotify_add_watch -> 0x17
            java-13752 [010] 180569.131432: sys_inotify_add_watch -> 0x18
            java-13752 [010] 180569.135184: sys_inotify_add_watch -> 0x19
            java-13752 [010] 180569.135616: sys_inotify_add_watch -> 0x1a
            java-13752 [010] 180569.139202: sys_inotify_add_watch -> 0x1b
            java-13752 [010] 180569.139622: sys_inotify_add_watch -> 0x1c
            java-13752 [010] 180569.149321: sys_inotify_add_watch -> 0x1d
            java-13752 [010] 180569.149717: sys_inotify_add_watch -> 0x1e
            java-13752 [010] 180569.156260: sys_inotify_add_watch -> 0x1f
            java-13752 [010] 180569.165739: sys_inotify_add_watch -> 0x20
            java-13752 [010] 180569.169937: sys_inotify_add_watch -> 0x21
            java-13752 [010] 180569.170296: sys_inotify_add_watch -> 0x22
            java-13752 [010] 180569.177402: sys_inotify_add_watch -> 0x23
            java-13752 [010] 180569.183846: sys_inotify_add_watch -> 0x24
            java-13752 [010] 180569.187312: sys_inotify_add_watch -> 0x25
            java-13752 [010] 180569.187802: sys_inotify_add_watch -> 0x26
            java-13752 [010] 180569.191314: sys_inotify_add_watch -> 0x27
            java-13752 [010] 180569.191781: sys_inotify_add_watch -> 0x28
            java-13752 [010] 180569.198126: sys_inotify_add_watch -> 0x29
            java-13752 [010] 180569.201667: sys_inotify_add_watch -> 0x2a
            java-13752 [010] 180569.209703: sys_inotify_add_watch -> 0x2b
            java-13752 [010] 180569.212063: sys_inotify_add_watch -> 0x2c
            java-13752 [010] 180569.214432: sys_inotify_add_watch -> 0x2d
            java-13752 [010] 180569.214729: sys_inotify_add_watch -> 0x2e
            java-13752 [010] 180569.216971: sys_inotify_add_watch -> 0x2f
            java-13752 [010] 180569.219159: sys_inotify_add_watch -> 0x30
            java-13752 [010] 180569.219450: sys_inotify_add_watch -> 0x31
            java-13752 [010] 180569.221780: sys_inotify_add_watch -> 0x32
            java-13752 [010] 180569.222029: sys_inotify_add_watch -> 0x33
            java-13752 [010] 180569.225990: sys_inotify_add_watch -> 0x34
            java-13752 [010] 180569.228548: sys_inotify_add_watch -> 0x35
            java-13752 [010] 180569.228797: sys_inotify_add_watch -> 0x36
            java-13752 [010] 180569.232822: sys_inotify_add_watch -> 0x37
            java-13752 [010] 180569.233054: sys_inotify_add_watch -> 0x38
            java-13752 [010] 180569.237234: sys_inotify_add_watch -> 0x39
            java-13752 [010] 180569.237551: sys_inotify_add_watch -> 0x3a
            java-13752 [010] 180569.243332: sys_inotify_add_watch -> 0x3b
            java-13752 [010] 180569.245901: sys_inotify_add_watch -> 0x3c
            java-13752 [010] 180569.246179: sys_inotify_add_watch -> 0x3d
            java-13752 [010] 180569.250486: sys_inotify_add_watch -> 0x3e
            java-13752 [010] 180569.250802: sys_inotify_add_watch -> 0x3f
            java-13752 [010] 180569.252945: sys_inotify_add_watch -> 0x40
            java-13752 [010] 180569.253189: sys_inotify_add_watch -> 0x41
            java-13752 [010] 180569.255402: sys_inotify_add_watch -> 0x42
            java-13752 [010] 180569.255661: sys_inotify_add_watch -> 0x43
            java-13752 [010] 180569.259566: sys_inotify_add_watch -> 0x44
            java-13752 [010] 180569.261640: sys_inotify_add_watch -> 0x45
            java-13752 [010] 180569.263669: sys_inotify_add_watch -> 0x46
            java-13752 [010] 180569.265819: sys_inotify_add_watch -> 0x47
            java-13752 [010] 180569.267893: sys_inotify_add_watch -> 0x48
            java-13752 [010] 180569.269967: sys_inotify_add_watch -> 0x49
            java-13752 [010] 180569.271976: sys_inotify_add_watch -> 0x4a
            java-13752 [010] 180569.272240: sys_inotify_add_watch -> 0x4b
            java-13752 [010] 180569.291990: sys_inotify_add_watch -> 0x4c
            java-13752 [010] 180569.292369: sys_inotify_add_watch -> 0x4d
            java-13752 [010] 180569.292726: sys_inotify_add_watch -> 0x4e
            java-13752 [010] 180569.293091: sys_inotify_add_watch -> 0x4f
            java-13752 [010] 180569.293420: sys_inotify_add_watch -> 0x50
            java-13752 [010] 180569.293749: sys_inotify_add_watch -> 0x51
            java-13752 [010] 180569.305760: sys_inotify_add_watch -> 0x52
            java-13752 [010] 180569.306204: sys_inotify_add_watch -> 0x53
            java-13752 [010] 180569.306665: sys_inotify_add_watch -> 0x54
            java-13752 [010] 180569.307042: sys_inotify_add_watch -> 0x55
            java-13752 [010] 180569.307385: sys_inotify_add_watch -> 0x56
            java-13752 [010] 180569.307724: sys_inotify_add_watch -> 0x57
            java-13752 [010] 180569.308032: sys_inotify_add_watch -> 0x58
            java-13752 [010] 180569.321561: sys_inotify_add_watch -> 0x59
            java-13752 [010] 180569.321968: sys_inotify_add_watch -> 0x5a
            java-13752 [010] 180569.322274: sys_inotify_add_watch -> 0x5b
            java-13752 [010] 180569.322552: sys_inotify_add_watch -> 0x5c
            java-13752 [010] 180569.322830: sys_inotify_add_watch -> 0x5d
            java-13752 [010] 180569.323106: sys_inotify_add_watch -> 0x5e
            java-13752 [010] 180569.323378: sys_inotify_add_watch -> 0x5f
            java-13752 [010] 180569.323635: sys_inotify_add_watch -> 0x60
            java-13752 [010] 180569.337109: sys_inotify_add_watch -> 0x61
            java-13752 [010] 180569.337452: sys_inotify_add_watch -> 0x62
            java-13752 [010] 180569.337779: sys_inotify_add_watch -> 0x63
            java-13752 [010] 180569.338094: sys_inotify_add_watch -> 0x64
            java-13752 [010] 180569.338379: sys_inotify_add_watch -> 0x65
            java-13752 [010] 180569.338660: sys_inotify_add_watch -> 0x66

Note the task and PID columns:

[email protected]:~# ps waux | grep java
root     13679 50.3  4.1 6393844 510320 pts/1  SNl  11:58   1:18 /usr/local/crashplan/jre/bin/java -Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx1024m -Djava.net.preferIPv4Stack=true -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0 -Dc42.native.md5.enabled=false -classpath /usr/local/crashplan/lib/com.backup42.desktop.jar:/usr/local/crashplan/lang com.backup42.service.CPService

The PID doesn’t always match up with the process that added the watch, in the example above, CrashPlan likely spawned a child process (PID: 13752, according to our trace) to add the inotify watches.

So now you know why this is happening, here is what you should do about it, First, to see what the currently configured limit is:

cat /proc/sys/fs/inotify/max_user_watches

It seems that the default limit for Ubuntu servers is 8192. To raise the limit, run the following as root:

sysctl -w fs.inotify.max_user_watches=32768

Or, to make the limit permanent, edit /etc/sysctl.conf and append the following line:


Then be sure to re-load the config file using the following command:

sysctl -p

The limit 32768 might be a bit high[**] so you may want a lower one depending on the available resources (RAM, CPU, etc.) of your machine. For reference, I use this configuration on production servers with 12GB RAM or more. YMMV.

To put things back to their default settings (defaults for Ubuntu anyway):

echo 0 > /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
echo 1 > /sys/kernel/debug/tracing/tracing_enabled

(On ubuntu, the default setting for /sys/kernel/debug/tracing/tracing_enabled is “1”)

Notes / Further reading:

Active Directory users aren’t able to update their password via SonicWall NetExtender

This seems to be a problem for a lot of people on the ‘net so I figured I’d document it here.

First, we need to make sure LDAP authentication is setup properly; take a look at the following screenshots (this assumes that you’ve already created a separate account for binding to the AD/LDAP database… in my examples below, this account is referred to as the “ldap bind” account.)

Security Settings

Security Settings (from https://www.fuzeqna.com/sonicwallkb/ext/kbdetail.aspx?kbid=5097)

Make to select 'LDAP' and not 'LDAP + Local Users' .

Make to select ‘LDAP’ and not ‘LDAP + Local Users’ .

Use the "display name" of the user account that you'd like to use for binding to the Active Directory server. Make sure that this account does NOT have Administrative privileges.

Use the “display name” of the user account that you’d like to use for binding to the Active Directory server. Make sure that this account does NOT have Administrative privileges.

Click "Import user groups" and select the group that contains the users who will should be granted SSLVPN access. Then set that group as the "Default LDAP User Group". I usually create a separate group for this purpose.

Click “Import user groups” and select the group that contains the users who will should be granted SSLVPN access. Then set that group as the “Default LDAP User Group”. I usually create a separate group for this purpose.

This is where you'll configure the locations of the OUs that contain Groups and Users who will have SSLVPN access.

This is where you’ll configure the locations of the OUs that contain Groups and Users who will have SSLVPN access.

Once you’ve done all of that, you’ll need to delegate the ability to change user passwords to your “ldap bind” account. This “sandboxes” the account; i.e., it allows the account to be able to change passwords without requiring administrative privileges. This way, the account only has the exact level of privilege it needs to do its job. Active Directory will allow you to use the Delegate Control wizard to delegate the Reset user passwords and force change at next logon task, but depending on which version of Windows Server you are running, this might not work as expected. In order to provide an LDAP Bind user with the ability to authenticate users and change passwords via NetExtender and Virtual Office, you can instead delegate access in the following manner:

  1. In Active Directory Users and Computers, right click on the desired OU and select Delegate Control
  2. Click Next.
  3. Click Add.
  4. Select a user or group, then click OK.
  5. Click Next.
  6. Select Create a custom task to delegate, and then click Next.
  7. Click Only the following objects in the folder, click to select the User objects check box, and then click Next.
  8. Click to select the General and the Property-specific check boxes.
  9. Click to select the Reset Password, Read pwdLastSet, and Write pwdLastSet check boxes in the Permissions box.
  10. Click Next, and then click Finish.

If for some reason it’s still not working, ensure that User objects are configured to Include inheritable permissions from this object’s parent, from the Advanced Security Settings page on each user object that needs to be able to update their password via NetExtender and/or Virtual Office. Note: You must have Advanced Features enabled in the view from within ADUC in order to see this page. To enable it, navigate to View and select Advanced Features. Now you’ll be able to see the Security tab of any Organizational Unit or Object.

Advanced Security Settings for [user object]

If you delegated the ability to change/reset passwords to your ldap bind account on the parent container of the target user objects, then each child user object must have this option selected. Alternatively, you could add the ldap bind account to the security settings on each user, manually.




Accessing Domain Resources When Connected To External VPN Via Windows 7 VPN Client

Recently one of my users was having trouble printing from remote workstations. In this scenario, the user needed to print some reports from a system that resided on a client’s network. She would start up the Windows 7 VPN client, provide credentials, and once connected to the client’s network through VPN, she would start the Microsoft Terminal Services Client and login to the system she needed to print reports from. During this time, the printers on our network (managed by a domain-wide printserver; mapped to workstations via GPO) would become inaccessible. The printers would “show up” on the remote side (as TS Session Printers) as they should, but they just would not accept print jobs.

After some troubleshooting, I found that this only happened when she was connected to the VPN. Whenever I tried to access the print server through a UNC path (i.e., \\printserver\printer ) a username/password box would appear, asking for her domain credentials — but here’s the kicker: the username field would be pre-filled with the username she used to login to the VPN. After some more digging, I came across this blog post and it immediately solved the problem. This was such a pain in the ass that I’ve decided to recreate the text of the resolution here, just in case that post should disappear for whatever reason:

  1. Locate the .pbk file (VPN session file) for the session you want to fix
    • Windows Vista/7: C:\Users\\AppData\Roaming\Microsoft\Network\Connections\Pbk
    • Windows XP: C:\Documents and Settings\All Users\Application Data\Microsoft\Network\Connections\Pbk
  2. Open the file in notepad and search for the following text: UseRasCredentials=1
  3. Change the “1” to a “0”, save, exit.

This tells the operating system not to rely on the RAS credentials that get cached upon initiating the VPN session. In Windows Vista / 7, this option is enabled by default; it wasn’t in Windows XP. I have yet to see/hear a good explanation for why it was changed.

Configure Windows Server 2008/2012 To Sync With Internet Time Servers

Question: There is no “Internet Time” tab in the date/time dialog box on Windows Server, is it still possible to configure the server to use NTP? If so, how?

Answer: Yes, as far as I know, you have to do this from the command prompt. Here’s how:

net stop w32time
w32tm /config /syncfromflags:manual /manualpeerlist:"time-a.nist.gov, time-b.nist.gov, time-c.nist.gov, time-d.nist.gov"
w32tm /config /reliable:yes
net start w32time

That will configure the time service to sync with the list of servers (time-*.nist.gov in the above example) and it also tells the server that it is a reliable time source that client machines on your domain can sync with. In other words, these steps configure the server as an NTP server in addition to configuring it to sync with Internet time servers. If you do not want that functionality, do not run the following command:

w32tm /config /reliable:yes

If you need to view the NTP configuration, type the following command from a prompt:

w32tm /query /configuration

This produces the following output:


EventLogFlags: 2 (Local)
AnnounceFlags: 5 (Local)
TimeJumpAuditOffset: 28800 (Local)
MinPollInterval: 6 (Local)
MaxPollInterval: 10 (Local)
MaxNegPhaseCorrection: 172800 (Local)
MaxPosPhaseCorrection: 172800 (Local)
MaxAllowedPhaseOffset: 300 (Local)

FrequencyCorrectRate: 4 (Local)
PollAdjustFactor: 5 (Local)
LargePhaseOffset: 50000000 (Local)
SpikeWatchPeriod: 900 (Local)
LocalClockDispersion: 10 (Local)
HoldPeriod: 5 (Local)
PhaseCorrectRate: 7 (Local)
UpdateInterval: 100 (Local)


NtpClient (Local)
DllName: C:\Windows\system32\w32time.dll (Local)
Enabled: 1 (Local)
InputProvider: 1 (Local)
AllowNonstandardModeCombinations: 1 (Local)
ResolvePeerBackoffMinutes: 15 (Local)
ResolvePeerBackoffMaxTimes: 7 (Local)
CompatibilityFlags: 2147483648 (Local)
EventLogFlags: 1 (Local)
LargeSampleSkew: 3 (Local)
SpecialPollInterval: 3600 (Local)
Type: NTP (Local)
NtpServer: time-a.nist.gov, time-b.nist.gov, time-c.nist.gov, time-d.nist.gov (Local)

NtpServer (Local)
DllName: C:\Windows\system32\w32time.dll (Local)
Enabled: 1 (Local)
InputProvider: 0 (Local)
AllowNonstandardModeCombinations: 1 (Local)

VMICTimeProvider (Local)
DllName: C:\Windows\System32\vmictimeprovider.dll (Local)
Enabled: 0 (Local)

If you are still having trouble responding to NTP requests after following these steps, make sure that there are no firewalls blocking udp/123. After you’ve checked your firewall, confirm that your NTP server responds by running the following command on a different Windows machine:

w32tm /stripchart /computer: /samples:2


w32tm /stripchart /computer: /dataonly /samples:2

In the following screenshot, I have run the two commands above, and then on the third execution, I attempt to query a server that does not exist, just so you can see what the output looks like if the client still can’t connect to your NTP server.

Querying an NTP server from another computer

Adding “Trusted Sites” to Internet Explorer, via the registry

Update! 5 February 2014 This can also be accomplished via GPO.

  • Open the group policy editor.
  • Create a new policy (or edit an existing policy.)
  • Navigate to:
    Computer Configuration/Administrative Templates/Windows Components/Internet Explorer/Internet Control Panel/Security Page/
  • The setting to add sites to the “Trusted Sites” zone is called “Site to Zone Assignment List”. Read the explanation in the “Help” box before configuring anything!
  • Then, to set configuration options for the “Trusted Sites” zone, you’ll want to navigate to the subdirectory/subkey titled “Trusted Sites Zone”. There you will find every setting that governs the behavior for that zone.

Original Article Follows

A while ago I needed to add a list of websites to the Internet Explorer’s “Trusted Sites” zone for multiple users, scattered across multiple terminal servers. IE’s “Enhanced Security Configuration” (ESC) is configured by default on windows terminal services and it’s normally a good idea to leave it intact.

However, this can have unintended consequences for users who require the use of websites that employ ActiveX, javascript, etc. because, by default, ESC does not allow those items to run. Sometimes, this means that the site in question will only be partially non-functioning. Other times, the entire site will be completely unusable. Furthermore, most users on terminal services have only a limited ability to actually modify the settings for an entire zone. Normally the best thing they can do is add the site to their trusted sites zone, if in fact the site is legitimate (i.e., “trusted”).

Originally, I explained to the users the steps involved in adding a site to their trusted sites, however many of the users used many of the same websites that other users were using. Also, new users needed to be trained on how to do this as well. Needless to say, it got very repetitive, very fast; so I came up with a “global” list of sites that can be trusted, and imported them to the registry on each terminal server. The list consisted of about 40+ sites, and I was able to generate the list mostly by exporting the following registry key:

HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\EscDomains

…from a few user accounts who had already added most of the sites to their trusted sites zone. After grepping out the duplicates (among other things), I had my list.

Now, I’m going to cover two ways of making this list of domains “globally trusted”—both of them involve writing to the following registry key:

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\EscDomains

Pay attention! This is not the same key as previously mentioned. This key resides in the ‘HKEY_LOCAL_MACHINE’ hive, whereas the previous key resides in the ‘HKEY_CURRENT_USER’ hive.

The first way is via the following visual basic script:

Option Explicit
Dim DomainArray(5), strComputer, strHTTP, strHTTPS
Dim dwordZone, regPath, objReg, counter, subkeyPath
Dim subkeyValue
Const HKEY_LOCAL_MACHINE = &H80000002
DomainArray(0) = "testdomain0.com"
DomainArray(1) = "testdomain1.com"
DomainArray(2) = "testdomain2.com"
DomainArray(3) = "testdomain3.com"
DomainArray(4) = "testdomain4.com"
strComputer = "."
strHTTP = "http"
strHTTPS = "https"
dwordZone = "2"
regPath = "SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings" &_
Set objReg = GetObject("winmgmts:{impersonationLevel = impersonate}!\\" & _
        strComputer & "\root\default:StdRegProv")
For counter = 0 to 4
        subkeyPath = regPath & DomainArray(counter)
        objReg.CreateKey HKEY_LOCAL_MACHINE,subkeyPath
        objReg.SetDWORDValue HKEY_LOCAL_MACHINE,subkeyPath,strHTTP,dwordZone
        objReg.SetDWORDValue HKEY_LOCAL_MACHINE,subkeyPath,strHTTPS,dwordZone

This script will insert ‘testdomain0.com’, ‘testdomain1.com’, […] into IE’s trusted sites zone when run on any machine. It must be run by an Administrator (or another user who has access to write to the HKEY_LOCAL_MACHINE registry hive), and the changes are global (to the machine).

The next way involves creating a “registry entries” (.reg) file:

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\EscDomains]
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\EscDomains\testdomain0.com]
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\EscDomains\testdomain1.com]
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\EscDomains\testdomain2.com]
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\EscDomains\testdomain3.com]
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\EscDomains\testdomain4.com]

Just like the previous script, this must also be run by a user with Administrator privileges and any changes will be global to all users on the machine.

(Of course, you would want to customize these snippets of code to suit your needs.)

For more information, please visit the following sites:

Internet Explorer Enhanced Security Configuration changes the browsing experience
Enhanced Security Configuration for Internet Explorer
Internet Explorer security zones registry entries for advanced users