Deploying Adobe Acrobat DC via Group Policy

This post assumes that you’ve already downloaded the Adobe Acrobat DC package from Adobe LWS and you’ve downloaded and installed the Acrobat Customization Wizard DC if not, go do those first.


  1. Once you’ve downloaded the package, double click it to extract its contents — do not run the installer after the package has been extracted.15-07-2015_000115-07-2015_0002
  2. Assuming you’ve extracted the package contents to C:\Users\Administrator\Desktop\Adobe Acrobat, you’ll have a directory within that directory, titled Adobe Acrobat, in addition to some other files that we won’t concern ourselves with for the purposes of this post:15-07-2015_0003
  3. Now start the Acrobat Customization Wizard DC:15-07-2015_0004
  4. Navigate to File -> Open Package and select the AcroPro.msi file.15-07-2015_0005
  5. Configure your deployment options. For example, enter the package’s serial number in the Serial Number field, under the Personalization Options category. Please refer to Adobe’s documentation for details on all of the available configuration options and their uses. One particularly frustrating issue I ran into had to do with the installer looking for Visual C++ x64 2013 Runtime (VC); if this package wasn’t installed, the installation would fail and the only thing logged in event viewer would be a group policy failure. The total lack of useful error messages made this issue a pain to overcome, but after reading every single option and description (more or less) on this page I eventually figured it out. I needed to set an attribute in the Property table via the Direct Editor function in the Customization Wizard.Add AttributeI had to add this attribute manually (just right-click anywhere in the attributes pane), as it does not exist in the Property table by default. The attribute name needed to be IGNOREVCRT64 and it’s value needed to be 1.IGNOREVCRT64You can probably guess at what this does but for the official explanation, refer to Adobe’s Documentation on Property attributes and values. The short explanation is that I needed this installation to succeed regardless of whether the target machine already had the Visual C++ x64 2013 Runtime package installed. As of this writing, here is the explanation, for posterity:

    Since Acrobat looks for Visual C++ x64 2013 Runtime (VC) by default, set IGNOREVCRT64 to 1 if it is not present AND not needed. IGNOREVCRT64 need only be used when all of the following are true:

    • During Acrobat installs on 64-bit machines
    • When Visual C++ x64 2010 SP1 Runtime is not installed
    • Installation is NOT done via setup.exe.
    • The following functionality is NOT needed:
      • Acrobat PDF Creation add-on (PDFMaker plugin) for Microsoft Office 64-bit applications (viz. Word, Excel, PowerPoint & Outlook) and
      • sending emails or resolving addresses via 64-bit Microsoft Outlook.

    Since Acrobat looks for Visual C++ x64 2013 Runtime (VC) by default, set to 1 if it is not present AND not needed. During non-setup.exe installs when VC is not present, installation behavior is as follows:

    • Command line installs abort if IGNOREVCRT64=1 is not passed.
    • UI installs display a dialog asking the user whether they would like to continue or cancel.
    • Setup.exe installs succeed. Installs never abort as the property is passed while spawning the MSI.

    For MSI installs, use Require64BitVC10RT in the Setup.ini file under the Startup section.

  6. Once you’ve configured all of your desired options, navigate to File -> Save Package. This will create a transform (.mst) file (… in addition to doing a few other things): 15-07-2015_0007
  7. To deploy this package, it will need to be accessible from the network. Create a network-accessible directory — something like:
    \\fileserver.domain.local\Deployed Files

    and copy the entire Adobe Acrobat directory — the one containing the transform file created in the previous step — to that directory.

  8. Open GPMC and create a GPO (or modify an existing GPO):
    15-07-2015_00102
  9. Right click on any blank space in the right panel and create a new package: 15-07-2015_0010
  10. Select Advanced deployment method:15-07-2015_0009
  11. Navigate to the network accessible location of the package and select it (note: it might take a while for any dialogs to appear after selecting the file … just be patient): 15-07-2015_0011
  12. Then add your transform (.mst) file in the Modifications tab of the package properties (the transform file will be in the same directory as the package file.) Then click OK: 15-07-2015_0012
  13. Bonus! Configure some administrative templates.
    1. First, download the templates from Adobe.
    2. Save them somewhere on your computer and unzip them.
    3. Open GPMC and edit the GPO you’d like to use with the templates.
    4. In the policy editor, navigate to Administrative Templates in either Computer or User configuration
    5. Right click on the Administrative Templates folder and select Add/Remove Templates Add/Remove Templates
    6. Navigate to the directory where you unzipped the templates and select the file named AcrobatDC.adm Select the .adm file AcrobatDC
    7. Then configure your options:Adobe "Computer Configuration" optionsAdobe "User Configuration" options

apcupsd pcnet and different subnets

Scenario:

  • Subnet A: 192.168.0.0/24
  • Subnet B: 10.10.10.0/24
  • No firewalls blocking communication between them

Problem: An apcupsd instance running on a machine residing on subnet “B” cannot communicate with the APC NMC that resides on subnet “A”

Solution (probably): Make sure to specify the port in your DEVICE declaration in /etc/apcupsd/apcupsd.conf:

DEVICE 192.168.0.5:username:password:3052

Snippet from the default apcupsd.conf:

# pcnet ipaddr:username:passphrase:port
# PowerChute Network Shutdown protocol which can be
# used as an alternative to SNMP with the AP9617
# family of smart slot cards. ipaddr is the IP
# address of the UPS management card. username and
# passphrase are the credentials for which the card
# has been configured. port is the port number on
# which to listen for messages from the UPS, normally
# 3052. If this parameter is empty or missing, the
# default of 3052 will be used.

The question is: Why does explicitly defining something that is supposed to be the default (even when left undefined), change the behavior of the program? I do not have an answer at present. Good luck though!

Event ID 4098 / 0x80070005 Access is denied when Copying files via Group Policy

Event ID 4098 logged in Event Viewer "Application" log.

Event ID 4098 logged in Event Viewer “Application” log.

This scenario is common enough — you’d like to copy a file to each user’s Desktop/My Documents/etc. without having to do it manually for each user. So you use Group Policy. You’ve done everything correctly (or so you think):

  • The file you’re trying to deploy has been shared and the GP preference item’s “Source file(s)” input box has been pointed to the file via the UNC path (not the local filesystem path) to the file.
  • If your GP Preference that copies the file resides in the “User Configuration” branch, you’ve ensured that the “Domain Users” group has read access to the directory that contains the file (NTFS permissions) and the Share.
  • If your GP Preference that copies the file resides in the “Computer Configuration” branch, you’ve ensured that the “Domain Computers” group has read access to the directory that contains the file (NTFS permissions) and the Share.
  • You’ve ensured that the policy has been linked to the correct OU (i.e., the OU that contains the “target” users or computers.)

But did you remember to specify the full path — including the filename in the “Destination File” input box?

You must include the destination filename, specifying the target directory by itself is not sufficient.

You must include the destination filename, specifying the target directory by itself is not sufficient.

Yeah… it happens to the best of us. But don’t beat yourself up over it. This is a problem that shouldn’t exist; it’s counter-intuitive and different from the way “copy” commands (which is basically what this is) normally work.

Problems configuring hylafax server on Ubuntu 12.04, 14.04

It feels wrong, writing an article about faxing in 2014 but here goes…

Getting HylaFAX to work on Ubuntu always seems to be a pain in the ass and today I’ve finally found out why. See the following bug reports:

https://bugs.launchpad.net/ubuntu/+source/uucp/+bug/255200

https://bugs.launchpad.net/ubuntu/+source/uucp/+bug/584787

All of my past attempts at getting HylaFAX to work with Ubuntu have always failed, causing me to have to switch to a different distro (usually CentOS) for HylaFAX servers. These two bug reports contain the answers.

The problem is twofold:

  1. First, permissions on /dev/ttyS* are incorrect by default (as far as HylaFAX is concerned, anyway — see the first bug report above.)
  2. Second, cu doesn’t seem to work correctly when trying to troubleshoot the problems caused by #1 (see the second bug report above.)

So here’s what you need to do in order to get a functioning HylaFAX installation [n1]:

  1. Install the necessary software:
    apt-get install cu hylafax-server
    
  2. Figure out which device node your modem is on:
    [email protected]:~# dmesg | grep ttyS
    [    0.684804] 0000:02:00.0: ttyS4 at I/O 0xda00 (irq = 21, base_baud = 115200) is a 16550A
    

You can see from the above that my modem is on /dev/ttyS4. Now before configuring HylaFAX you will need to know the class capabilities of your modem. To find that out, do the following [n2]:

  1. Create a new file: /etc/uucp/port that contains the following lines:
    #
    # Description for the TCP port - pretty trivial. DON'T DELETE.
    #
    port TCP
    type tcp
    
  2. Change the permissions on that file:
    chmod 644 /etc/uucp/port
    chown root.uucp /etc/uucp/port
    
  3. Make sure that the modem has the correct permissions on it’s device node [n3]:
    chmod 666 /dev/ttyS4
    
  4. Connect to your modem and query its capabilities [n4]:
    [email protected]:~# cu -l /dev/ttyS4
    Connected.
    at+fclass=?
    0,1,1.0,2,2.0,2.1
    OK
    ~[localhost].
    Disconnected.
    

    We see from the above output our modem supports all available classes, 0 – 2.1.

  5. Now all you need to do is run faxsetup and then faxaddmodem. I won’t cover that here (see links below) [r1] but a word of advice: when faxaddmodem asks for the “class” of your modem, make sure you have read, and you understand the characteristics of each available class before choosing [r3].

After you’ve run faxsetup and faxaddmodem [r1], you may want to configure HylaFAX to send all incoming faxes to an e-mail address as PDF attachments. You’ll need to install an MTA for this (I use postfix) and configure it as a satellite system so that your fax server will route mail through your mail server (did I mention that this assumes you already have access to a production mail server?) This is pretty trivial on Ubuntu; after running apt-get install postfix you are prompted for the type of configuration you want — satellite in this case, and once you’ve selected that option, you are prompted for the hostname/IP address of the mail server you plan on using to relay your faxes (i.e., the IP address of your production mail server.) You’ll need to ensure that that mail server is configured to relay mail coming from your fax server [r2]. Then, there are some HylaFAX specific things you’ll need to configure after you have a functioning MTA:

  1. Find out where your aliases are stored:
    [email protected]:~# postconf alias_database
    alias_database = hash:/etc/aliases
    
  2. Add the following line to that file:
    FaxDispatch:	[email protected]
    
  3. Update the aliases hash:
    [email protected]:~# newaliases
    
  4. Create a file; /var/spool/hylafax/etc/FaxDispatch, that contains the following lines:
    FILETYPE=pdf;
    TEMPLATE=en;
    SENDTO="[email protected]";
    
  5. Restart hylafax [n5]:
    [email protected]:~# /etc/init.d/hylafax restart
     * Stopping HylaFAX faxq                                                                                                [ OK ] 
     * Starting HylaFAX syncing directories...
    

Now go cable up your fax server and send some faxes to it!

Notes:

[1.]Installing cu is optional; you should only need it if you do not know the capabilities of your modem.

[2.] This also involves addressing the problems caused by the two bug reports mentioned at the beginning of this article.

[3.]If these permissions do not seem to work, try rebooting. If that does not work, try the following permissions:

chmod 600 /dev/ttyS4
chown uucp.dialout /dev/ttyS4

[4.] You may not see your typing echo back to the terminal; just continue typing -or- copy and paste the following in its entirety:

cu -l /dev/ttyS4
at+fclass=?
~~

Then at the [hostname] prompt, just hit the period “.” key and wait a second or two.

Make sure you are not in /var/spool/hylafax/etc when you restart HylaFAX! Part of the start/stop/restart process involves unmounting /etc/hylafax from /var/spool/hylafax/etc and then remounting it. You’ll know if you mess this up because you’ll see the following output, repeated until you hit CTRL+C:

umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /var/spool/hylafax/etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

Resources:

[1.] Using faxsetup and faxaddmodem:
http://www.hylafax.org/content/Handbook:Basic_Server_Configuration:Faxsetup
http://www.hylafax.org/content/Handbook:Basic_Server_Configuration:Faxaddmodem

[2.] Configure postfix to relay mail from other clients:
http://www.postfix.org/BASIC_CONFIGURATION_README.html#relay_from

[3.] Fax Classes:
http://en.wikipedia.org/wiki/Fax#Class

tail: “inotify resources exhausted” and/or “inotify cannot be used, reverting to polling: Too many open files”

If you happen across either of these messages while tailing a logfile:

  • tail: inotify resources exhausted
  • tail: inotify cannot be used, reverting to polling: Too many open files

… And you have CrashPlan installed[*], then you probably have too low a limit on the number of inotify.max_user_watches. I only mention CrashPlan because this seems to be fairly common with CrashPlan on Linux. This could happen for a variety of reasons actually so to find out what is causing it, do the following:

echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
echo 1 > /sys/kernel/debug/tracing/tracing_enabled

Those two commands will enable you to “watch” inotify_add_watch events. To actually watch them, wait a few minutes after enabling, and then:

cat /sys/kernel/debug/tracing/trace

You should see some output similar to this:

[email protected]:~# cat /sys/kernel/debug/tracing/trace | more
# tracer: nop
#
#           TASK-PID    CPU#    TIMESTAMP  FUNCTION
#              | |       |          |         |
            java-13752 [010] 180569.026114: sys_inotify_add_watch -> 0x1
            java-13752 [010] 180569.038573: sys_inotify_add_watch -> 0x2
            java-13752 [010] 180569.039368: sys_inotify_add_watch -> 0x3
            java-13752 [010] 180569.044214: sys_inotify_add_watch -> 0x4
            java-13752 [010] 180569.051454: sys_inotify_add_watch -> 0x5
            java-13752 [010] 180569.052107: sys_inotify_add_watch -> 0x6
            java-13752 [010] 180569.059542: sys_inotify_add_watch -> 0x7
            java-13752 [010] 180569.060265: sys_inotify_add_watch -> 0x8
            java-13752 [010] 180569.060760: sys_inotify_add_watch -> 0x9
            java-13752 [010] 180569.068002: sys_inotify_add_watch -> 0xa
            java-13752 [010] 180569.068549: sys_inotify_add_watch -> 0xb
            java-13752 [010] 180569.082694: sys_inotify_add_watch -> 0xc
            java-13752 [010] 180569.089735: sys_inotify_add_watch -> 0xd
            java-13752 [010] 180569.093624: sys_inotify_add_watch -> 0xe
            java-13752 [010] 180569.094271: sys_inotify_add_watch -> 0xf
            java-13752 [010] 180569.098156: sys_inotify_add_watch -> 0x10
            java-13752 [010] 180569.098794: sys_inotify_add_watch -> 0x11
            java-13752 [010] 180569.105731: sys_inotify_add_watch -> 0x12
            java-13752 [010] 180569.109630: sys_inotify_add_watch -> 0x13
            java-13752 [010] 180569.119702: sys_inotify_add_watch -> 0x14
            java-13752 [010] 180569.123390: sys_inotify_add_watch -> 0x15
            java-13752 [010] 180569.127319: sys_inotify_add_watch -> 0x16
            java-13752 [010] 180569.127801: sys_inotify_add_watch -> 0x17
            java-13752 [010] 180569.131432: sys_inotify_add_watch -> 0x18
            java-13752 [010] 180569.135184: sys_inotify_add_watch -> 0x19
            java-13752 [010] 180569.135616: sys_inotify_add_watch -> 0x1a
            java-13752 [010] 180569.139202: sys_inotify_add_watch -> 0x1b
            java-13752 [010] 180569.139622: sys_inotify_add_watch -> 0x1c
            java-13752 [010] 180569.149321: sys_inotify_add_watch -> 0x1d
            java-13752 [010] 180569.149717: sys_inotify_add_watch -> 0x1e
            java-13752 [010] 180569.156260: sys_inotify_add_watch -> 0x1f
            java-13752 [010] 180569.165739: sys_inotify_add_watch -> 0x20
            java-13752 [010] 180569.169937: sys_inotify_add_watch -> 0x21
            java-13752 [010] 180569.170296: sys_inotify_add_watch -> 0x22
            java-13752 [010] 180569.177402: sys_inotify_add_watch -> 0x23
            java-13752 [010] 180569.183846: sys_inotify_add_watch -> 0x24
            java-13752 [010] 180569.187312: sys_inotify_add_watch -> 0x25
            java-13752 [010] 180569.187802: sys_inotify_add_watch -> 0x26
            java-13752 [010] 180569.191314: sys_inotify_add_watch -> 0x27
            java-13752 [010] 180569.191781: sys_inotify_add_watch -> 0x28
            java-13752 [010] 180569.198126: sys_inotify_add_watch -> 0x29
            java-13752 [010] 180569.201667: sys_inotify_add_watch -> 0x2a
            java-13752 [010] 180569.209703: sys_inotify_add_watch -> 0x2b
            java-13752 [010] 180569.212063: sys_inotify_add_watch -> 0x2c
            java-13752 [010] 180569.214432: sys_inotify_add_watch -> 0x2d
            java-13752 [010] 180569.214729: sys_inotify_add_watch -> 0x2e
            java-13752 [010] 180569.216971: sys_inotify_add_watch -> 0x2f
            java-13752 [010] 180569.219159: sys_inotify_add_watch -> 0x30
            java-13752 [010] 180569.219450: sys_inotify_add_watch -> 0x31
            java-13752 [010] 180569.221780: sys_inotify_add_watch -> 0x32
            java-13752 [010] 180569.222029: sys_inotify_add_watch -> 0x33
            java-13752 [010] 180569.225990: sys_inotify_add_watch -> 0x34
            java-13752 [010] 180569.228548: sys_inotify_add_watch -> 0x35
            java-13752 [010] 180569.228797: sys_inotify_add_watch -> 0x36
            java-13752 [010] 180569.232822: sys_inotify_add_watch -> 0x37
            java-13752 [010] 180569.233054: sys_inotify_add_watch -> 0x38
            java-13752 [010] 180569.237234: sys_inotify_add_watch -> 0x39
            java-13752 [010] 180569.237551: sys_inotify_add_watch -> 0x3a
            java-13752 [010] 180569.243332: sys_inotify_add_watch -> 0x3b
            java-13752 [010] 180569.245901: sys_inotify_add_watch -> 0x3c
            java-13752 [010] 180569.246179: sys_inotify_add_watch -> 0x3d
            java-13752 [010] 180569.250486: sys_inotify_add_watch -> 0x3e
            java-13752 [010] 180569.250802: sys_inotify_add_watch -> 0x3f
            java-13752 [010] 180569.252945: sys_inotify_add_watch -> 0x40
            java-13752 [010] 180569.253189: sys_inotify_add_watch -> 0x41
            java-13752 [010] 180569.255402: sys_inotify_add_watch -> 0x42
            java-13752 [010] 180569.255661: sys_inotify_add_watch -> 0x43
            java-13752 [010] 180569.259566: sys_inotify_add_watch -> 0x44
            java-13752 [010] 180569.261640: sys_inotify_add_watch -> 0x45
            java-13752 [010] 180569.263669: sys_inotify_add_watch -> 0x46
            java-13752 [010] 180569.265819: sys_inotify_add_watch -> 0x47
            java-13752 [010] 180569.267893: sys_inotify_add_watch -> 0x48
            java-13752 [010] 180569.269967: sys_inotify_add_watch -> 0x49
            java-13752 [010] 180569.271976: sys_inotify_add_watch -> 0x4a
            java-13752 [010] 180569.272240: sys_inotify_add_watch -> 0x4b
            java-13752 [010] 180569.291990: sys_inotify_add_watch -> 0x4c
            java-13752 [010] 180569.292369: sys_inotify_add_watch -> 0x4d
            java-13752 [010] 180569.292726: sys_inotify_add_watch -> 0x4e
            java-13752 [010] 180569.293091: sys_inotify_add_watch -> 0x4f
            java-13752 [010] 180569.293420: sys_inotify_add_watch -> 0x50
            java-13752 [010] 180569.293749: sys_inotify_add_watch -> 0x51
            java-13752 [010] 180569.305760: sys_inotify_add_watch -> 0x52
            java-13752 [010] 180569.306204: sys_inotify_add_watch -> 0x53
            java-13752 [010] 180569.306665: sys_inotify_add_watch -> 0x54
            java-13752 [010] 180569.307042: sys_inotify_add_watch -> 0x55
            java-13752 [010] 180569.307385: sys_inotify_add_watch -> 0x56
            java-13752 [010] 180569.307724: sys_inotify_add_watch -> 0x57
            java-13752 [010] 180569.308032: sys_inotify_add_watch -> 0x58
            java-13752 [010] 180569.321561: sys_inotify_add_watch -> 0x59
            java-13752 [010] 180569.321968: sys_inotify_add_watch -> 0x5a
            java-13752 [010] 180569.322274: sys_inotify_add_watch -> 0x5b
            java-13752 [010] 180569.322552: sys_inotify_add_watch -> 0x5c
            java-13752 [010] 180569.322830: sys_inotify_add_watch -> 0x5d
            java-13752 [010] 180569.323106: sys_inotify_add_watch -> 0x5e
            java-13752 [010] 180569.323378: sys_inotify_add_watch -> 0x5f
            java-13752 [010] 180569.323635: sys_inotify_add_watch -> 0x60
            java-13752 [010] 180569.337109: sys_inotify_add_watch -> 0x61
            java-13752 [010] 180569.337452: sys_inotify_add_watch -> 0x62
            java-13752 [010] 180569.337779: sys_inotify_add_watch -> 0x63
            java-13752 [010] 180569.338094: sys_inotify_add_watch -> 0x64
            java-13752 [010] 180569.338379: sys_inotify_add_watch -> 0x65
            java-13752 [010] 180569.338660: sys_inotify_add_watch -> 0x66
--More--

Note the task and PID columns:

[email protected]:~# ps waux | grep java
root     13679 50.3  4.1 6393844 510320 pts/1  SNl  11:58   1:18 /usr/local/crashplan/jre/bin/java -Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx1024m -Djava.net.preferIPv4Stack=true -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0 -Dc42.native.md5.enabled=false -classpath /usr/local/crashplan/lib/com.backup42.desktop.jar:/usr/local/crashplan/lang com.backup42.service.CPService

The PID doesn’t always match up with the process that added the watch, in the example above, CrashPlan likely spawned a child process (PID: 13752, according to our trace) to add the inotify watches.

So now you know why this is happening, here is what you should do about it, First, to see what the currently configured limit is:

cat /proc/sys/fs/inotify/max_user_watches

It seems that the default limit for Ubuntu servers is 8192. To raise the limit, run the following as root:

sysctl -w fs.inotify.max_user_watches=32768

Or, to make the limit permanent, edit /etc/sysctl.conf and append the following line:

fs.inotify.max_user_watches=32768

Then be sure to re-load the config file using the following command:

sysctl -p

The limit 32768 might be a bit high[**] so you may want a lower one depending on the available resources (RAM, CPU, etc.) of your machine. For reference, I use this configuration on production servers with 12GB RAM or more. YMMV.

To put things back to their default settings (defaults for Ubuntu anyway):

echo 0 > /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
echo 1 > /sys/kernel/debug/tracing/tracing_enabled

(On ubuntu, the default setting for /sys/kernel/debug/tracing/tracing_enabled is “1”)

Notes / Further reading: