The Inner Circle

As many of you know I work at one of the smaller universities in the UK, we have about 12000 students. On our network we have 6500 student residences and over 15000 ports throughout the campus for staff and students. On average we pull 400mbs and push 150mbs to our JANET regional network (which we also run ;).

Now and again various department, such as Physics, like to stress our links by swapping mass quantities of research data, last week they helped us average 980mbs incomming for a good four hours (we have 1gbps). I think it’s important to stress that we are an academic university, we encourage research, we don’t place many restrictions on what staff computers can do on our network. At the end of the day we provide a high speed network to be used.

However based upon experience I think many people are under the impression we are clueless about what happens on our network, or are incapable of looking. Which, I’m afraid is wishful thinking. Now that doesn’t mean we are consiously aware of every single action that takes place on the network, anyone who knows how much data 400mbs is will tell you it’s impossible to comprehend, but that doesn’t mean we don’t have a record. So what do we have…

Proxy/Server Logs
This is by far the most obvious one, every client PC records logs, so by rule you should assume our servers do. Which they do. All web traffic is proxied through one of four http proxies, these record your IP address, the date/time and what webpage you visited. These are kept for a significant amount of time, and we are often asked deal with http based abuse (usually to Wikipedia). Our services that require log on credentials always record who logged in from where and when.

Network Identification
Many people think that if they plug a laptop into a random network port get an IP address that they have never had before and isn’t assigned to them that we don’t know what it is. This isn’t true, everything that can be plugged into our network has a number that is unique (or it’s ment to be). We can simply look up this number and see what other IP addresses it’s been, and then look through the above mentioned server logs for usernames.

Network Authentication
This one is up and coming, we intend to implement a system that allows us who can plug into our network and what level of access they get when they do. I won’t go into how this works because we haven’t decided yet, suffice to say every device will be marked against a person and they will be responsible for it.

NetFlow
Here’s the big one for us, we fully know that the internet is not just webpages. There’s things like eDonkey and BitTorrent for doing peer2peer, there’s IRC, ICQ or Jabber for talking. You can watch movies in QuickTime or RealPlayer that come in live. You can make phone calls with Skype, play games on an Xbox or PC. We can’t proxy these things, so how do we know that they have happened.

Every ISP worth it’s salt, be they Pipex, BT, Zen, Bulldog, Tiscali, JANET, the university all use the same method (and more) to ensure they have records. All large scale routers, we’re talking the ones worth £50k+ not at £40 Belkin or Buffalo (although OSS firmware often has it for them) are capable of recording Network Flows.

Network Flows are records of communications on the internet, they don’t include the content of the communication, they just confirm that it took place. They record your IP address, the IP address of the server, the port numbers involved, the date/time, the number of packets, the number of bytes and a few other more technical pieces of information.

Yeasturday we recorded 1.5Gb of NetFlows which equates to about 30 million records, which our system thinks is 150Gb of data and that’s it not running at 100%.

Mirror Ports
Under the RIPA and the computing AUP the university has rights to ensure that it’s network is being used for purposes that do not break it’s AUP and diagnose network problems. Because of this we have ports coming off of our equipment that provides a complete copy of all data flowing through it. These are plugged into servers that can be used to search through the data and identify different flows in real time. We can also uses these mirror ports for running intrusion detection systems.

What else?
Having written this I try to think whatelse we do to ensure that we have some form of a record of what users do, I’m sure there is something, but I can’t think at the moment. It is exceptionally important to point out that access to all these different pieces of data is controlled heavilly to ensure that it can not be misused, and is not available to the majority of staff at the university.

For example mirror ports and network flows are only available to networking team leaders and the security team, server logs are usually only available to the admin team of that server, network identification records are available to all members of the technical groups as they do not contain sensitive or identifying information. Everyone with access to this type of data takes it very seriously as misuse is a serious incident and potentially illegal.

As I personally administrate the mirror servers and network flow server I can say that I take every care to ensure that the server is kept secure and access is limited to those that have a geniune need.

I hope this has opened a few peoples minds, but not frightened or worried them about big brother. It’s also worth writting that any views stated are my own and not thoose of my employer.

D.

So, we have our nice Soekris 4801 AP, and the world is good…. or is it? Not for me…

My partner is a Mac user, which normally isn’t a problem. Mac’s tend to be fairly good at following standards and them being build on a combination of Mach and FreeBSD, it can’t be a bad thing. It does however appear they either have something non-standard or are following the standard to the letter with no budging either way.

It occasionally takes three goes at getting the laptop to associate with my FreeBSD AP, and even then once connected it looses packets and generally sees a week signal. Now I know not what the problem is, sadly I don’t have access to any 802.11 debugging software/equipment at work, we don’t do out of the ordinary wireless.

I was always suspect of the RALink chipset on the Belkin cards, as the issue I found with sis0 going offline is reported to be a ral driver issue (who knows I barely understand PCI and kernels at that level). So to disprove the Soekris I got the spare Belkin card and put it in my core FreeBSD router, a proper PCI V2.2 compliant machine.

Same issue, phew, which means it’s either the RALink chipset or FreeBSD. After scrambling around my house looking for other PCI wireless cards I find another card, also RALink but a much more modern chipset, damn, not supported in FreeBSD 6.2, it’s only just in Current. Hmmm, where on a Thursday with no car, can I get another PCI card…. PC World.

I don’t like admitting I shop there, but for instant things, providing you can fend off the vultures, it’s doable. Not pleasant, but still :). Before I went I looked up ath(4) and found a list of cards that are supported, looked through the list, discounting makes like Cisco that PC World wouldn’t have, trying to find out if there where any gotcha’s, like naughty suppliers changing chipsets.

I looked at ath(4) because I’d seen the MadWiFi project that provides some very funky features with this type of chipset (for example Virtual AP’s, i.e. more then one ssid :), and I hope one day FreeBSD will get some of these things.

I go to said establishment, which I shall try not to mention again. Look around their supply of PCI wireless cards, they have a few. Most of them I can instantly cross off, either they are RALink or they are a chipset FreeBSD has no support for.

Like a shining grail above castle Anthrax, I see a Netgear WG311T, excitement and temptation. I have it on my list as an ath(4) chipset, but only for version 1. Hmm, that’s a problem, is it version 1? I find one that some vandal had already opened, so ahum, I decided to have a further look. Turns out this is actually a WG311TGE… GE? Never seen a GE before.

I look at the back of the PCB and see WG311TGEV1H3, well it claims it’s a version one. Time to be really naughty, out comes the Leatherman and a few gentle lifts of the RF shield on the card, ping, oooooh Atheros. I’ll be having that!

I took that and another one that looking at another code on the boxes appeared to be revision 2. Checked out, of course explaining that this one box was already open, and if it didn’t work, I’d be bringing it back. The attendant of course agreed assuming it was pre-owned stock.

Once I got home I took both the RF shield off of the cards, the PCB’s where different, but both are ath(4). Bonus. Nothing like having spares. Off came the lid of my router and in went the ath(4) card. Just like the RALink, it just worked after loading kernel modules.

Then I tried it with the Mac… hmmm, signal strength is slightly better, and there are less errors but it’s still not perfect. Not like the Intel Centreno card in my work laptop. So let’s play around with ifconfig some more. I previously knew about setting “media auto mode 11g mediaopt hostap” and the “pureg” option, and then I found after executing “ifconfig -vk ath0″ txpower… oooh what’s that then? I’ve seen maxtxpow, but it didn’t seem to do anything.

Instantly tried “ifconfig ath0 txpower 100″, after reading back the value it went to 60. But still slightly higher then it was before. Tried the Mac again, it was better, much better. Infact usable, okey 1% packet loss in 22000 packets, but I’m assuming the issue here is signal strength. I don’t know if it’s receive or transmit though.

After a bit better positioning, it seems to be down to less then 1%, which seems to be fine. The ath(4) driver also has different code you can choose to tweak how the card decides what speed to send the data at, I haven’t played with this, but it might also help.

Previously people have asked for the commands I’m using in rc.conf, so I present it. If FreeBSD ever get’s Virtual AP support I’m sure I’ll have examples for that, I’m tempted to look into how it works, but again if it’s to much chipset work, I fear I’ll be lost. I just suck at hardware programming.

ifconfig_sis0="inet 10.10.10.50 netmask 255.255.255.0 up"
ifconfig_vlan0="vlandev sis0 vlan 50 up"
ifconfig_ath0="ssid Alastria channel 9 media auto mediaopt hostap mode 11g pureg
txpower 60 nwkey xxxx hidessid up"
ifconfig_bridge0="addm vlan0 addm ath0 up"

A hint is to use “nwkey ” this automatically expands to “wepmode on wepkey deftxkey 1″, it’s also worth before setting it up, doing a “ifconfig ath0 list ap” and finding out what channels other people are using. There are only three non overlapping channels in WiFi I’m told, 1, 7 and 11 (, although I think 7 may be questionable).
Now that things are working happily for the Mac I can concentrate on making a web interface for the AP, as I already have PHP, SQLite and Lighttpd on my compact flash card. FWIW I’m still only using 16mb out of the 64mb CF card. Why should things be big?

Bellow is the output of ifconfig for comparison.

%ifconfig -vk ath0
ath0: flags=8943 <UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
ether 00:14:6c:72:7c:e5
media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap>
status: associated
ssid Alastria channel 9 (2452) bssid 00:14:6c:72:7c:e5
authmode OPEN privacy ON deftxkey 1
wepkey 1:104-bit <xxxxxxxxxxxxxxxxxxxxxxxxxx> tx+rx+def
powersavemode OFF powersavesleep 100 txpowmax 37 txpower 60
rtsthreshold 2346 mcastrate 1 fragthreshold 2346 pureg protmode CTS
-wme burst ssid HIDE apbridge dtimperiod 1 bintval 100

It’s also worth noting the serial numbers and TA(!?) of the cards.

One: TA: 100-11024-01R10 SN: 1573583902875

Two: TA: 100-11024-01R12 SN: 1573633V01805

That may help someone, it may not, both are sold as WG311T on the box, but inside the card says WG311TGE. Both are Atheros inside and both have been working as APs for me.
TTFN,

D.

I searched about the web for information on how to do this and didn’t find anything obvious (all examples where far to complex). My home network (as I’ve said before) is overly complex, it’s designed to mirror many of the different techniques that I use at work. To that end, I have a management network which is heavily locked down, which provides me access to the various router and switch management interfaces.

I had a problem though, in order to reach my management IP on my router I had to trunk a vlan to it, which worked fine. But because I already had a default route I couldn’t route back managment traffic easily (without many many different routes) back to my network, so I could only ever configure the router from my core house router. This sucked.
I had heard about VRFs at work in relation to MPLS, I thought that it might help. Sure enough it’s done what I want. I think I’m best demonstrating this with route print outs and the config rather then words.

ala-2620-ro01#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is not set
ala-2620-ro01#sh ip route vrf mgmt

Routing Table: mgmt
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is 10.10.10.1 to network 0.0.0.0

10.0.0.0/24 is subnetted, 1 subnets
C 10.10.10.0 is directly connected, FastEthernet0/0
S* 0.0.0.0/0 [1/0] via 10.10.10.1
ala-2620-ro01#sh ip route vrf inet

Routing Table: inet
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is 0.0.0.0 to network 0.0.0.0

88.0.0.0/29 is subnetted, 1 subnets
C xxx.xxx.xxx.xxx is directly connected, FastEthernet0/0.10
62.0.0.0/32 is subnetted, 1 subnets
C 62.3.83.5 is directly connected, Dialer0
S* 0.0.0.0/0 is directly connected, Dialer0
Config:

version 12.3
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname ala-2620-ro01
!
logging buffered 4096 informational
logging console informational
!
no aaa new-model
ip subnet-zero
ip cef
!
no ip domain lookup
ip domain name alastria.lan
ip name-server 10.10.10.1
!
ip vrf inet
!
ip vrf mgmt
!
ip multicast-routing vrf mgmt
ip multicast-routing vrf inet
!
ip audit po max-events 100
!
username peter privilege 15 secret 5 xxxx
username monitor secret 5 xxxx
!
interface ATM0/0
no ip address
no ip proxy-arp
no atm ilmi-keepalive
dsl operating-mode auto
pvc 0/38
encapsulation aal5mux ppp dialer
dialer pool-member 1
!
interface FastEthernet0/0
ip vrf forwarding mgmt
ip address 10.10.10.10 255.255.255.0
no ip proxy-arp
duplex auto
speed auto
!
interface FastEthernet0/0.10
encapsulation dot1Q 10
ip vrf forwarding inet
ip address xxx.xxx.xxx.xxx 255.255.255.248
ip pim sparse-dense-mode
no snmp trap link-status
!
interface Dialer0
ip vrf forwarding inet
ip unnumbered FastEthernet0/0.10
ip pim sparse-dense-mode
encapsulation ppp
ip route-cache flow
ip igmp unidirectional-link
dialer pool 1
no cdp enable
ppp authentication chap callin
ppp chap hostname xxxx
ppp chap password 0 xxxx
!
ip http server
ip http authentication local
no ip http secure-server
!
ip classless
!
ip route vrf mgmt 0.0.0.0 0.0.0.0 10.10.10.1
ip route vrf inet 0.0.0.0 0.0.0.0 Dialer0
!
logging facility local0
logging source-interface FastEthernet0/0
logging 10.10.0.1
!
access-list 10 permit 10.10.10.0 0.0.0.255
access-list 10 deny any
!
snmp-server community public RO 10
snmp-server enable traps tty
!
line con 0
login local
line aux 0
line vty 0 4
login local
transport input ssh
line vty 5 15
login local
transport input ssh
!
ntp clock-period 17179815
ntp source FastEthernet0/0
ntp server 10.10.10.1
!
end

At home I run a rather complex network, I take the chance with my home network to mirror and experiement with some of the concepts that we use at work. I have a heavily subnet’d/vlan’d network, each server sits on a /30 point to point, all network devices have a management IP and if possible it’s on the management vlan, the VoIP network is again seperate.

I decided to create two main client networks, these are known as the “inner lan” and “outer lan”, in the “inner lan” we have the trusted periment computers this is the least restricted subnet, the router still has a default to deny policy, but there are many more open ports. The “outer lan” on the otherhand has absolutly no access to other networks.

The idea is that untrusted computers are put on the “outer lan” and they must either by VPN/IPsec tunnels, proxys or a closed garden except a log in before allowed through. So what does this have to do with a 11g access point?

I intend to run open access points so that anyone can log onto the AP, once they have they are placed in the “outer lan” and must validate in order to gain access to the rest of the network or internet.

Consumer access points have come a long way in the five years that I have been using them, however there is something quite critical to my plan that prevents me from using one, commercial access points support this feature, but you’re looking at shelling out over £200 (at least).

The problem is that all APs are configurable over IP, this means their IP is reachable once logged onto wireless lan. I trust my “outer lan” router to be secure, however there are numerious issues with various APs that could be exploited.

So what do I need? I need to be able to put the APs IP on a seperate VLAN and trunk that back to the feeding switch. This relativly simple function isn’t available until you start spending serious money. I put my mind to it and woundered how I could do it on the cheap.

I already had a Soekris single board computer, it is a 266Mhz 586 processor with 128Mb of RAM, there’s a compact flash slot, a mini-pci slot, and importantly to me a full size 3.3V PCI slot.

I did some research about 802.11g PCI cards and my favourite OS (FreeBSD) and found that the Belkin F5D7000 PCI card was supported by the “ral” driver. First step was to build a small version of FreeBSD that would be able to run on the Soekris, I found the guide for miniBSD was very help full and without tuning makes an image of about 13Mb (with tweaking, lighttpd and php5-fastcgi I have managed 15Mb).
So I took the Soekris appart and on my work bench ran the SBC with the PCI card and was happily suprised to find that it was capable of acting as access point.

This was fine until disaster struck, occasionally when transfering data (probably due to the speed of moving data between the CPU and the network cards), the onboard nic would stop recieving data. It turns out that if the sis chip overflows it’s DMA recieve buffer the card shuts down the RX engine. I did some more research and developed a patch that stoped the network card from stopping.

I now have a fully working access point and with the help of the ral, sis, if_bridge and vlan drivers I have been able to build an AP I would have had to pay of £200 for.

So how much did it actually cost… the Soekris 4801-50 cost £115, the Belkin F5D7000 was £17 and a 64Mb CF card was free with a digital camera of the past. You maybe thinking that £132 is not that cheap, but because this is running FreeBSD you can do anything you want with it still, plug in ng_flow and generate netflows for auditing, if WPA is your thing then hostapd will help you.

If anyone wants any hints of what I did then leave me a comment (w/ email) and I’ll get back to you.