Wireless Dilemma and Kubuntu 9.04 Network Manager….

While upgrading my kid’s computer and installing the web proxy and filter (see article titled “SquidGuard Blacklists…“), I ran across a real problem.  Wireless would start only after a user logged into their desktop, so the system had no IP address until then.  However, without an IP, Dansguardian would fail to start.  I tried scripting the problem away, essentially waiting indefinitely until a periodic check showed an IP address in use and then starting the services, but this did not work.  I played around with making an init script under /etc/init.d and using “update-rc.d” to create the proper sym links.  This also did not work.  I tried manually defining the wireless network using /etc/network interfaces and creating a /etc/wpa_supplicant.conf file.  This did not work.

It was then I remembered a server I had built at work, using Ubuntu-9.04, in which I had stripped off all of the GUI/desktop stuff, leaving a bare-bones server instead.  It worked fine on the network, and did not have Network Manager installed.  Looking in the init script folder under /etc/init.d, I found a NetworkManager service, so I made it non-executable (“sudo chmod -x /etc/inint.d/NetworkManager“), and ran “sudo update-rc.d -f NetworkManager remove” to get rid of the startup links.  After that, the wireless network started on boot just fine, with no need for user interaction, and the services for the proxy and filters started flawlessly (I added them into /etc/network/interfaces).

So, Network Manager was stepping all over /etc/network/interfaces.  Not anymore.  I could have removed the package, but other packages will then be removed, and I don’t want that.

For someone having trouble with their manual wireless setup, here are my scrubbed /etc/network/interfaces and /etc/wpa_supplicant.conf files:

/etc/network/interfaces:

auto lo
iface lo inet loopback

auto wlan0
iface wlan0 inet dhcp
wpa-conf /etc/wpa_supplicant.conf

# added 10-18-09 for proxy filter
pre-up iptables-restore < /etc/iptables.rules
post-up /usr/local/squid/sbin/squid
post-up /usr/local/dansguardian/sbin/dansguardian
post-down iptables-save -c > /etc/iptables.rules

/etc/wpa_supplicant.conf:

network={
ssid="myssid"
proto=RSN
key_mgmt=WPA-PSK
pairwise=CCMP TKIP
group=CCMP TKIP
psk="my-key-phrase"
}

This is for a WPA2 wireless setup (SSID and passphrase are bogus, of course).  Hope this helps someone.

SquidGuard Blacklists…

Here is a listing of some sites that have actively managed blacklists freely available for non-commercial download:

Shalla Secure Services
Blacklists UT1
MESD Blacklists (not sure how current this one is)

Anyway, I updated the getlists.sh script from the HOWTO – Child-Proofing Internet Access on Kubuntu article. It was failing because squidGuard kept not finding files and going into emergency mode when run with “-C all” to build databases. By also running it with the -d option, I was able to see where it was failing. The Norway site was not permitting the blacklist download to occur, so I found these other sites and wrote that into the script. By doing that and adjusting my squidguard.conf file (commented out the “not_ok” ACL block), as well as by creating files that it could not find (copied ok/domains.db to ok/domains and adult/very_restrictive_expressions to adult/expressions and porn/expressions), the script now ran without errors to completion.

The script is updated here and on the linked article.
getlists.sh (pdf file)

HOWTO – Child-Proofing Internet Access on Kubuntu

[UPDATED 10-18-2009 – Numerous old typos fixed, several new typos added, syntaxes corrected, updates made for newer versions of stuff, better instructions, cooler errors, and even a little more attention to detail paid.]

[CREDIT for getlists.sh goes to Step By Step – www.maynidea.com. Thank you for this script, and sorry it took so long to put this credit in.]

This article is a revision of this post. It has been adapted for use on Kubuntu 8.04. I got a lot of info from this link here. Another excellent resource is here (PDF). As always, YMMV. This is a long and involved post – be prepared to take an afternoon, and to work on that degree from Google. But when you are done, you will have a powerful transparent-proxy-content-filter-porn-stomper. No charge.

1. Download the following (there may be newer versions, but definitely need db-2.7.7):

I checked these versions against the repositories, and except for the db-2.7.7, these are still fairly current. The version of iptables I am using is 1.3.8. For this, I prefer installing from tarballs, even though this means they will not get updates. The main advantages I see to this approach are that you can more directly control where they go in the file system (making them easier to troubleshoot and remove), and updates to packages might cause feature/config file breakage, whereas these ensure a static environment. Unfortunately, I cannot upload the actual tarballs for use, so either find these versions in an archive, or brace yourself for an adventure in configuration differences.

2. Unpack the downloaded files:

  • tar xvfz db-2.7.7.tar.gz
  • tar xvfj squid-2.6.STABLE5-20061110.tar.bz2
  • tar xvfz dansguardian-2.9.8.0.tar.gz
  • tar xvfz squidGuard-1.2.0.tar.gz

3. Check that you don’t already have squid, squidGuard, or dansguardian already installed, and that you have iptables installed. Adept Manager is an easy way to find out. Check that you do not already have a squid group and user. If you do not, then pick a group ID between 1 and 999 to use for the squid group:

  • more /etc/group | grep -i squid <is there a squid group?>
  • more /etc/passwd | grep -i squid <is there a squid user?>
  • more /etc/login.defs | grep -i UID_MIN <what is the lowest user ID? anything below this is a system account, and will not get a home directory by default, which is a good thing – so pick something lower than UID_MIN>
  • more /etc/group | grep <number below UID_MIN> <is the group ID you picked already in use? If so, keep picking one until you find a number not in use.>

4. As root (sudo -s), make user and group. The “groupadd -r squid” command is out – this would have made a system account. The new command syntax is shown below instead.

  • groupadd -g <number you picked> squid
  • useradd -u <number you picked> -g squid -d /var/spool/squid -s /bin/false -r squid

5. When making firewall rules (below), I kept getting the error “iptables: No chain/target/match by that name” until I discovered that I did not have the ipt_owner.ko module available to be loaded (on my current version of 2.6.31.4, it is called “xt_owner”). Issue an “updatedb” command, followed by “locate _owner.ko” to see if you have it for your kernel version. If you have it, see if it is loaded – “lsmod | grep -i _owner“. I ended up compiling a new kernel from 2.6.26.2 to 2.6.28.5 (to get some other features I wanted, not just for the module), and ensuring the owner module was built (“make oldconfig” and “make menuconfig” steps of this post, under the networking section). Once I had that module, I was good to go with matching packets by owner.

Make menuconfig (need ncurses libraries installed: libncurses5-dev and libncursesw5-dev; helpful to have ncurses-term packages installed):
“Networking Support —>
Networking Options —>
Network Packet Filtering Framework (Netfilter) —>
Core Netfilter Configuration —>”

  • (M) Netfilter connection tracking support (NF_CONNTRACK)
  • (M) Transparent proxying support (EXPERIMENTAL) (NETFILTER_TPROXY)
  • (M) “TPROXY” target support (EXPERIMENTAL) (NETFILTER_XT_TARGET_TPROXY)
  • (M) “owner” match support (NETFILTER_XT_MATCH_OWNER)

REMEMBER: If you upgrade your kernel to a new version and use a proprietary video driver (ATI or NVIDIA), set your xorg.conf driver to “vesa” BEFORE you reboot. Reboot on the new kernel, log into the console (so as not to start any window manager or x session), and upgrade your video driver (update xorg.conf to reflect the new driver). Then either reboot, or just start your window manager normally.

6. Make BerkelyDB – must be 2.x version, not newer, not older:

  • cd db-2.7.7/dist/
  • ./configure
  • make
  • make install

7. Make squid v.2-6 (NOTE – To have SSL, I needed to install the libcurl4-openSSL-dev package. Otherwise, “make” generated this error: “../include/md5.h:14:2: error: #error Cannot find OpenSSL headers” ):

  • cd squid-2.6.STABLE5-20061110/
  • ./configure --enable-icmp --enable-delay-pools --enable-useragent-log --enable-referer-log --enable-kill-parent-hack --enable-cachemgr-hostname=hostname --enable-arp-acl --enable-htcp --enable-ssl --enable-forw-via-db --enable-cache-digests --enable-default-err-language=English --enable-err-languages=English --enable-linux-netfilter --disable-ident-lookups --disable-internal-dns
  • make
  • make install

It is located in /usr/local/squid/.

8. Make squidGuard v.1.2:

  • cd squidGuard-1.2.0/
  • ./configure
  • make
  • make install

Default install is in /usr/local/bin/.

9. Make dansguardian v.2.9.8:

  • cd dansguardian-2.9.8.0/
  • mkdir /usr/local/dansguardian
  • ./configure --prefix=/usr/local/dansguardian --with-proxyuser=squid --with-proxygroup=squid --enable-email=yes
  • FOR EMBEDDED URL WEIGHTING AND OTHER FEATURES: ./configure --prefix=/usr/local/dansguardian --with-proxyuser=squid --with-proxygroup=squid --enable-email=yes --enable-pcre=yes (this last option is CPU intensive; turn on in dansguardianf1.conf)
  • make
  • make install

It is located in /usr/local/dansguardian/.

If you get an error during the configure part like this: “configure: error: pcre-config not found!“, install the libpcre++-dev package.
When using GCC 4.3, I got errors of “error: ‘strncpy’ was not declared in this scope“. The fix was found on GCC 4.3 Release Series – Porting to the New Tools. Basically, for each such error, go to the file referenced under the src folder and add the line #include (cstring) (replace parentheses with angle brackets).

10. Make and configure squid directories:

  • mkdir /usr/local/squid/var/cache
  • chown -R squid:squid /usr/local/squid/var
  • chmod 0770 /usr/local/squid/var/cache
  • chmod 0770 /usr/local/squid/var/logs

11. Make and configure squidGuard directories (see getlists.sh for reference):

  • mkdir /usr/local/squidGuard
  • mkdir /usr/local/squidGuard/log
  • chown -R squid:squid /usr/local/squidGuard/log
  • chmod 0770 /usr/local/squidGuard/log
  • mkdir /var/log/squidguard
  • touch /var/log/squidguard/squidGuard.log
  • touch /var/log/squidguard/ads.log
  • touch /var/log/squidguard/stopped.log
  • chown -R squid.squid /var/log/squidguard
  • mkdir /var/lib/squidguard
  • mkdir /var/lib/squidguard/db
  • mkdir /var/lib/squidguard/db/blacklists
  • mkdir /var/lib/squidguard/db/blacklists/ok
  • mkdir /var/lib/squidguard/db/blacklists/porn
  • mkdir /var/lib/squidguard/db/blacklists/adult
  • mkdir /var/lib/squidguard/db/blacklists/ads
  • chown -R squid:squid /var/lib/squidguard

12. Configure dansguardian directories:

  • chown -R squid:squid /usr/local/dansguardian/var/log
  • touch /var/lib/squidguard/db/blacklists/porn/domains_diff.local
  • touch /var/lib/squidguard/db/blacklists/porn/urls_diff.local

13. Edit and copy squid configs from respective source directories:

  • cp squid.conf /usr/local/squid/etc/squid.conf
  • sample squid.conf settings:
    • http_port 127.0.0.1:3128 transparent
    • icp_port 0
    • htcp_port 0
    • redirect_program /usr/local/bin/squidGuard
    • cache_effective_user squid
    • cache_effective_group squid
    • acl all src 0.0.0.0/0.0.0.0
    • acl manager proto cache_object
    • acl localhost src 127.0.0.1/255.255.255.255
    • acl to_localhost dst 127.0.0.0/8
    • acl allowed_hosts src 192.168.12.0/255.255.255.0
    • acl SSL_ports port 443
    • acl Safe_ports port 80 21 443 # http ftp https
    • ##acl Safe_ports port 21 # ftp
    • ##acl Safe_ports port 443 # https
    • ##acl Safe_ports port 1025-65535 # unregistered ports
    • acl CONNECT method CONNECT
    • acl NUMCONN maxconn 5
    • acl ACLTIME time SMTWHFA 7:00-21:00
    • deny_info ERR_ACCESS_DENIED_TIME ACLTIME
    • #http_access allow manager localhost
    • #http_access deny manager
    • http_access deny manager all
    • http_access deny !Safe_ports
    • http_access deny CONNECT !SSL_ports
    • http_access allow localhost ACLTIME
    • http_access deny NUMCONN localhost
    • #http_access allow allowed_hosts
    • http_access deny to_localhost
    • http_access deny all
    • http_reply_access allow all
    • #icp_access allow allowed_hosts
    • #icp_access allow all
    • icp_access deny all
    • visible_hostname localhost

Edit squid.conf and set up time based access, to prevent late night surfing (add the following lines):

  • acl ACLTIME time SMTWHFA 7:00-21:00 (add to the ACL section)
  • http_access allow localhost ACLTIME (add to the http_access section)

14. Edit and copy squidGuard configs from respective source directories:

  • cp squidGuard.conf /usr/local/squidGuard/squidGuard.conf
    • change ip gateway address in squidGuard.conf

15. Edit and copy dansguardian configs from respective source directories:

  • cp dansguardia*.conf /usr/local/dansguardian/etc/dansguardian/
  • sample dansguardian.conf settings:
  • sample dansguardianf1.conf settings:
    • groupmode = 1
  • copy getlists.sh (it is posted as a PDF – copy the text to a shell script) to /usr/local/bin
  • [UPDATED 10-18-2009 with more current blacklist sites]

16. Make the firewall rules (iptables commands may appear wrapped in two lines):

  • iptables -t nat -A OUTPUT -s 127.0.0.1 -d 127.0.0.1 -p tcp --dport 3128 -j ACCEPT (without this rule, dansguardian may fail with the error: “Error connecting to parent proxy”)
  • iptables -t nat -A OUTPUT -p tcp --dport 80 -m owner --uid-owner squid -j ACCEPT
  • iptables -t nat -A OUTPUT -p tcp --dport 3128 -m owner --uid-owner squid -j ACCEPT
  • iptables -t nat -A OUTPUT -p tcp --dport 80 -m owner --uid-owner exemptuser -j ACCEPT (change exemptuser)
  • iptables -t nat -A OUTPUT -p tcp --dport 80 -j REDIRECT --to-ports 8080
  • iptables -t nat -A OUTPUT -p tcp --dport 3128 -j REDIRECT --to-ports 8080

It is a good idea to do this part *after* compiling and installing, as these rules will get in the way if you need to install a package (like libcurl4-openSSL-dev). If this happens, Adept Manager will abruptly crash (leaving you to find and remove the lock files), and apt-get install will fail with a connection refused error. Just rerun the rules above, but replace the -A with a -D to delete them. Get your packages, install your software, and reapply the firewall rules.

17. Save and apply the firewall settings permanently (visit Iptables HowTo – Community Ubuntu Documentation for details):

  • sudo sh -c "iptables-save > /etc/iptables.rules"
  • sudo nano /etc/network/interfaces
    • pre-up iptables-restore < /etc/iptables.rules
    • post-down iptables-save -c > /etc/iptables.rules

18. Start or restart services as needed:

  • /usr/local/squid/sbin/squid -z (first-time config)
  • /usr/local/squid/sbin/squid -N -d 1 -D (test squid, kill when working fine)
  • /usr/local/squid/sbin/squid (this also runs squidGuard from “/usr/local/bin/squidGuard”)
  • /usr/local/dansguardian/sbin/dansguardian
  • /usr/local/bin/getlists.sh (you may have to kill this – it hangs after displaying the line “adult/usage”)
  • /usr/local/squid/sbin/squid -k reconfigure
  • /usr/local/dansguardian/sbin/dansguardian -Q

The squid test revealed that I was missing a custom file: “errorTryLoadText: ‘/usr/local/squid/etc/errors/ERR_ACCESS_DENIED_TIME': (2) No such file or directory”. So, I copied it from “/usr/local/squid/etc/errors/English/ERR_ACCESS_DENIED”, and “edited” it in vi for a little access-denied humor. Never miss a chance to have a spot of fun! After that, squid worked fine.

Dansguardian kept failing with “Error connecting to parent proxy”, until I edited iptables with “iptables -t nat -I OUTPUT 1 -s 127.0.0.1 -d 127.0.0.1 -p tcp --dport 3128 -j ACCEPT"
(to place it as the first output rule on the nat table). Then DG worked fine.

The script hung and had to be killed. I confirmed everything was finished by checking the last file date-time-stamp against the date-time-stamp it displays right after it is run. So if the DTS displayed was “20090214185211”, and the DTS returned with “ls -l /var/lib/squidguard/db/blacklists/porn/stats/20090214185211_stats” was more recent, say “2009-02-14 18:53″, then you can be sure it is finished. Or you can just use “lsof” and look for the getlists.sh process. That is probably smarter.

[UPDATED 10-18-2009]
The script hung because a.) I could not download from the Norway site and b.) “squidguard -C all” from the getlists.sh script was not finding files and went into emergency mode, apparently a place it can hide and whimper silently. Forever. I ran instead “squidguard -d -C all” and discovered it was failing to find certain files, which I just created or copied into existence. This quieted squidguard down and let it finish. Almost – I also commented out the “not_ok” ACL block in the squidguard.conf file, since I am not using it. Details are on this article concerning the updated blacklist script “getlists.sh”: SquidGuard Blacklists…

19. Set up a mailer for notifications (here is a link for assistance):

  • using postfix, point it to your mailserver.isp.domain
  • postfix needs /etc/postfix/transport and /etc/postfix/generic
  • dansguardian.conf calls it with ‘sendmail -t' command
  • for non-authenticated use, do not set ‘by user = on’ in dansgaurdianf1.conf

20. Post-install testing and tweaking:

  • Test with browser as different users – should be transparent proxy surfing now, works with lynx as well (“su - <username>, lynx, G, http://www.playboy.com” should get either Playboy for an approved user or the dansguardian access denied page for a restricted user.)
  • Check if your system emails you violations.
  • Be sure to update your startup files (/etc/init.d/ or your rc.local) to ensure everything starts when the computer is booted.
  • When you are ready, reboot, and check again with lynx as different users.

I have been working on this all day. I have not yet gotten email to work, and am not sure I need to – maybe I’ll just check the logs instead. So, hope this helps, and good luck.

Time for a beer.

Follow

Get every new post delivered to your Inbox.