FreeBSD Tips & Tricks
In case of attacks, you can fiddle up with the following values:
- net.inet.tcp.msl (on my machine, 10000. default is 30000)
- net.inet.tcp.keepidle (default 10000)
- net.inet.tcp.keepintvl (default 7500)
- net.inet.tcp/udp.blackhole (turn on for DoS)
- net.inet.tcp.tcbhashsize (push up to a reasonable value)
For a better performance, you should mess up with:
- kern.ipc.somaxconn (my machine = 4096, default. 128)
- kern.ipc.maxsockets
- net.inet.ip.intr_queue_maxlen
- kern.maxfiles (65535, 16424 as default)
- vfs.vmiodirenable (set to 1, 0 is the default)
- net.inet.tcp.sendspace (see tuning(7) for more explanations)
- net.inet.tcp.recvspace
- option NMBCLUSTERS in the kernel (check how many clusters you use with netstat -mb -- don't overtune it, on my busiest webservers the number of used clusters never went above 2256, so 8192 should be enough for all servers), and, of course, maxusers. Well, that era has ended. Right now I discovered a
2827/16384/16384 mbuf clusters in use (current/peak/max)
on a webserver, so I'm gonna bump it to 32k. Also, I'm going to reduce the net.inet.tcp.sendspace from 32k to 16k, because the web traffic means a lot of small files. In case you find out you're just about to run out of NMBCLUSTERS (ex. 6301/8100/8704 mbuf clusters in use (current/peak/max) -- on one of my servers), in case of a webserver, and cannot recompile a new kernel with a bumped NMBCLUSTERS, then set KeepAlive to off in your Apache, and this will save you some clusters by removing many FIN_WAIT_2 connections (More here).
- kern.ipc.shm_use_phys -- turn to 1 if the main application of your server uses shared memory, it has improved the activity of my web server.
- netstat -f inet can give you valuable informations. For example, a lot of connections with Send-Q != 0 means that your server is storing datas into the mbufs, because the uplink is saturated or (more often) the clients are too 'slow' to receive datas. Non-null values for Recv-Q, on the other hand, means that your server is too slow in serving the requests, which increase the number of mbufs for incoming connections.
- net.inet.tcp.msl -- take it down from the default 30,000 to something like 10,000 or even less if you notice too many TIME_WAIT connections in netstat -f inet
- net.inet.tcp.inflight_enable -- for bandwidth delay limiting (TCP connections). Read more about in tuning(7).
Firewalling with IPF gave me quite some problems, mostly related to the state table. The customers experienced some broke connections (browser hanging forever when loading up a page), even though the channel was not full (bandwidth-wise). What you can do in a situation like this is play with:
- net.inet.ipf.fr_tcpidletimeout=7200 (I like to leave this higher, though, because it kills my idle ssh sessions on the servers as well -- and I hate logging in each other hour or so).
- net.inet.ipf.fr_tcpclosewait=120
- net.inet.ipf.fr_tcplastack=120
- net.inet.ipf.fr_tcptimeout=240
- net.inet.ipf.fr_tcpclosed=60
- net.inet.ipf.fr_tcphalfclosed=300
- net.inet.ipf.fr_udptimeout=90
- net.inet.ipf.fr_icmptimeout=35
More about this values here.
You can check how many states are active by looking into the output of ipfstat -s (active). I experienced values growing from 0 to approx. 4000 and then 0 again, which meant the state table got full, and was resetted. You can also carefully increment the number of states in '/usr/include/netinet/ip_state.h' (IPSTATE_SIZE and IPSTATE_MAX -- in my case IPSTATE_MAX was set to 4013, which made the table reset like I said before, at a value approx. equal to 4000). You have to set these two values at resonable values (not too high, don't overtune!), they need to be prime values, and IPSTATE_MAX should be approx. 70% of IPSTATE_SIZE. More infos can be found here.
One other thing that one might do is remove any unnecessary 'keep state's from the firewall configuration. For example, Apache communicates with the clients on port 80 exclusively, so if you 'pass all from any to $my_host port = 80', then you don't need keep states.
As about the NAT using IPFilter, you might consider defining LARGE_NAT in src/contrib/ipfilter/ip_nat.h and src/sys/contrib/ipfilter/netinet/ip_nat.h.
One other problem that I had, having the same source, was that FTP transfers of a zillion+ files would just stall from time to time (after about 100 transferred files). Reason was that the state table was becoming full, and was 'cleaned up' by the kernel, which meant lost state. Freeing the unneeded 'keep states' from the firewall rules, and twaking the parameters above made this problem dissapear as well.
A very nice feature of FreeBSD's kernel is also the DEVICE_POLLING kernel option, which basically means that the system will not treat any interrupt coming from the network cards independently, but rather 'poll' the devices at certain intervals of times. That saves a lot of system activity. You might also consider tweaking with the "option HZ", and also enabling 'kern.polling.enable' and 'kenr.polling.user_frac'. Unfortunately, DEVICE_POLLING works only with certain NICs, but I've experienced very good results with the Intel EtherExpress (fxp). You can see the performance on some snapshots of my firewall here.
Special settings:
- In order to run ipf and ipfw on the same machine (ipf for firewall, ipfw for traffic shaping), you can do the following:
ipf -f a_file, where a_file contains something like: "pass out quick proto tcp from x.x.x.x to y.y.y.y port = z flags S keep state"
ipfw add pipe 10 ip from x.x.x.x to y.y.y.y
ipfw pipe 10 config bw 10Kbit/s queue 50KBytes
Security issues:
This section wants to be a short list of tips & tricks for building up the most secure system out of your *BSD box. And it's possible that these advices will help you even if you're not using *BSD.
First, some dry theory: all we want to do is make it harder for the intruder to do any harm to our system. Also, we want to build up a way through which we can restore the functionality of the system (clean it up?) after an intrusion. For example, in most cases the intruder will replace basic utilities like 'ls' or 'ps' with it's own version. Thus, she/he can run different utilities/daemons on our system, without our knowledge (and very hard to detect). So, if we cannot count on 'ls' or 'ps', the only choice would probably be to reinstall everything from the scratch.
- First of all, make sure all the software on your system has been updated to an 'acceptable' version (from a security point of view). So consider a 'cvs up' of the /usr/src directory, and a 'make buildworld/buildkernel/installkernel/installworld' afterwards. Details can be found at http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html.
- After the installation process has finished succesfully, remove all the unecessary services running. Yeah, even consider removing the SSH daemon (see below). Check out all the open ports (netstat -an, and a nmap -sT from a remote machine), and double-check you need all the services listening on the opened ports!
- SSH: consider removing it completely, esspecially after all the bugs uncovered this summer (2002). Of course, when the machine is remote, and there is no other way to reach it, all you can do is remove all the unnecessary 'features' like 'PermitRootLogin yes', which come turned off in most of the cases (counter-example: OpenBSD, which has this feature turned on by default, which is really messy), but it doesn't hurt to double-check... Make sure you also trick the logging facilities of SSH and syslogd, to log the login attempts (be them succesfull or not), and of course to read these logs periodically. UnfortuNAtely, the "SyslogFacility" filed in "/etc/ssh/sshd_config" on FreeBSD comes with the default "AUTH", but there is no entry for "auth" in "/etc/syslog.conf". So somebody could try for months to discover your password by brute force, without having any mention in the logs about that. Also, consider setting up a firewall for your site, and filtering the SSH port from any location except for the hosts you really DO USE to log in into the respective machine.
- In the next step we need to build up barriers which will (hopefully! :) stop the intruder from polluting the basic system utilitites, even if she/he has obtained root access. Keeping this in mind, I suggest 'chflags' to 'schg' (the immutable flag) all the basic utilites, that you change only when upgrading the system (e.g.: "/bin", "/boot", "/etc" -- in the rare case you don't add users/change settings to often to your machine, "/sbin", "/usr/bin", "/usr/games", "/usr/include", "/usr/lib", "/usr/libdata", "/usr/libexec", "/usr/sbin", "/usr/share", etc, recursively). Of course, it doesn't make any sense to change the flags of these directories if the intruder can revert to the "no"-flags, which means that you should also consider increasing the kern.securelevel to at least "1". This technique guarantees that the intruder won't be able to change any of the base system binaries with hacked ones, except if he reboots the system into a lower securelevel (-1). Even in this case, there are few things the sysadmin can do, things involving periodic maintenance. On of these things is to create periodic checksums for the base system files on the server. For example, on might create a weekly list of md5 checksum for all the binaries in "/bin", "/sbin", "/usr/bin", etc., and for the configuration files in "/etc", and later compare these checksums with the checksums of the binaries/configuration files on a compromised system. The modified files will obviously be very easily discovered. This can be transformed into a weekly task anyways, where the comparison between these lists can be done automagically by your favourite cron script.
- Pretty interesting kernel module for system firewalling: http://cerber.sourceforge.net/.
Things that aren't really related to FreeBSD, but are security advices in general
I have decided to write this section after I've read this bugtraq mail about how to read any file on a disk using php and mysql. So, I'm gonna treat this subject first:
- Secure PHP in such a way as to be unable to read files on the disk, outside the DocumentRoot of the web server. This can be done using the 'open_basedir' in PHP. You can set this on a Directory/VirtualHost entry in Apache as well, which is even wiser, because you can refine the security settings.
- As about the MySQL part involved, make sure you drop all the "File" privileges for all the users that don't need them.
- Of course, the best approach is the hardcore one, meaning that you should consider very seriously chrooting the webserver to some directory, or running it into a jail system. This becomes less dependent on PHP/Apache, and more dependent on the actual OS, which has certain advantages (in a nutshell, makes it a lot harder to exploit).
Hehe, new record: 7:53PM up 11 days, 7:36, 2 users, load averages: 712.00, 366.41, 154.87
Found a nice page which treats about the same things here, you can check it out here.