These kernel modules are used to enhance network security by implementing a flexible kernel-level firewall. As with other aspects of the Linux kernel, default settings are often insufficient for a busy Ceph cluster. If your organization's policies permit it, you may blacklist these all together to keep them from loading. It's still prudent to raise their limits as a fallback option, as even blacklisted modules have a way of slipping back in. There is a connection table maintained by nf_conntrack that may default to as low as 65536. We suggest half a million as an ample value for OSD nodes hosting 24 4TB OSDs. Extremely dense nodes may require an even larger setting:

net.netfilter.nf_conntrack_max=524288
net.nf_conntrack_max=524288

Your kernel may use one or both of these names. Raising these will consume megabytes of additional kernel memory; on modern systems, this is trivial.

Below is an Ansible playbook to unload and remove iptables and nf_conntrack along with their dependencies.

# Unload and blacklist kernel modules related to iptables and nf_conntrack
# ref: https://goo.gl/aQFI8d
#
# Usage: ansible-playbook -e target=hostname rmmod.yml
# It is ok for some of the modules to fail during removal if
# they are not loaded. these are ignored.
- name: ensure we are applying to ceph server nodes
  assert:
    that: "'ceph_mon' in group_names or 'ceph_osd' in group_names or 'ceph_rgw' in group_names or 'ceph_aio' in group_names"
- name: stop and disable iptables
  service:
    name: iptables
    enabled: no
    state: stopped
- name: remove nat, conntrack modules. order here is important.
  command: rmmod {{ item }}
  with_items:
   - iptable_nat- nf_nat_ipv4
   - nf_nat
   - nf_conntrack_ipv4
   - nf_defrag_ipv4
   - nf_conntrack_proto_gre
   - xt_CT
   - nf_conntrack
   - iptable_filter
   - iptable_raw
   - ip_tables
  ignore_errors: true
- name: do not load conntrack on boot
  file: path=/etc/sysconfig/modules/ip_conntrack.modules state=absent
- name: do not load conntrack_proto_gre on boot
  file: path=/etc/sysconfig/modules/nf_conntrack_proto_gre.modules state=absent
- name: blacklist the modules to ensure they are not loaded on reboot
  copy:
    owner: root
    mode: 0644
    dest: /etc/modprobe.d/conntrack.conf
    content: |
      blacklist nf_conntrack
      blacklist nf_conntrack_ipv6
      blacklist xt_conntrack
      blacklist nf_conntrack_ftp
      blacklist xt_state
      blacklist iptable_nat
      blacklist ipt_REDIRECT
      blacklist nf_nat
      blacklist nf_conntrack_ipv4
      blacklist nf_conntrack_proto_gre
      blacklist xt_CT
      blacklist iptable_raw
      blacklist ip_tables

This playbook was designed for RHEL7.2 systems using an Ansible inventory file with certain hostgroup definitions. Your site practices, Linux distribution, and kernel release version will require adjustments to the inventory file and the lists of modules.

Every Ceph admin (and every sysadmin) has a favorite set of tunables and values to set, and there can be controversy over best practice. The names and effects of settings, as well as their defaults vary by kernel and Linux distribution release. Those we present here are based on our experiences. Your mileage, as they say, may vary, and you are encouraged to research what's right for you. The archives of the ceph-users mailing list are an especially rich hunting ground.

In Chapter 19, Operations and Maintenance, we learned mechanisms to configure myriad Ceph behavioral and tuning settings. We changed values both in the configuration file read at startup and dynamically in running daemons by injection. Those settings dovetail into those we describe in this chapter to maximize the stability and performance of your Ceph deployments.