Weather Station

Installed a Fine Offset Electronics WH1081 Weather Station on my roof the other day. Purchased the device from ebay for $85 and $25 delivered.

The station consists of the following sensors:

• Thermo-hydro transmitter
• Wind speed
• Wind direction
• Rain gauge

The console is a touch screen LCD panel. Apart from the LCD being difficult to read due to being too light (needs a contrast setting) it works reasonably well. The best part about it is that it has a USB connection for plugging into a PC.

To go with the USB conneciton the station comes with a software package called 'EasyWeather' which is functional and maintains a log with various graphs showing historical data. I wasn't too fussed on it though since it did things like give false readings and has really bad memory leaks.

I tried Cumulus, which is another weather station application that can upload results to a FTP site for Internet access. However it doesn't recognise the WH1081 natively and instead relies upon EasyWeather to gather statistics - not good for the above reasons of instablity and memory leaks.

Currently I am using the Linux console version of Weather Display. I like it because its a no fuss application with no frills and supports uploading results to Weather Underground natively. However it doesn't support 64bit at all, even when using IA32 libraries. This made me install 32bit Ubuntu (CLI only) within a VirtualBox - had to use the closed source version for the USB support. This is on my Mythbuntu 8.04 HTPC too, so the weather console sits on top of the TV in the loungeroom which worked out well.

You can see my weather data on Weather Underground here.

Network upgrade

And the upgrades continue.

The network here is expanding something chronic so I needed something that could push the vlans harder. It's basically replacing the local Mikrotik/WRAP1-1 Router and Asus GigaX2024 L2 switch with a single Asus GigaX3112 L3 switch. It certainly tidied up the rack by removing two switches and a stack of patch leads.

Now I have two Asus 2024 L2 switches to stick in Admissions and the Crocosium. This will give me a gigabit vlan trunk to the locations and allow me to create some more subnets to reduce some of this needless traffic off the main networks. I'm trying to have a L2 managed switch on the end of every fibre link to get some flexibility into the network and get things into this decade...

I'm not all that impressed with the Asus network kit so far. It's okay for the price but it's buggy as hell and the 3112 tends to crash due to kernel panics and reboot due to buffer overflows or memory errors. I'm hoping future firmware updates will come and fix things. Not that I had much more luck with Netgear and Linksys stuff. There's a reason why Cisco can charge so much.

The campus wireless network is almost completed. Awaiting a cable run from the Machinery shed to the Conference center to install the AP on the roof there. Also need to install the 12th AP at the Tiger Temple to complete the 'ring of coverage' - full wireless coverage of the safari shuttle track and nearby walkways and buildings. Will put off any more expansion until the Hotel is built.

Terminating fibres

Been busy terminating a few new fibre runs throughout the Zoo. This particular run goes from the Taj (Crocosium) through to the Dingo Diner and then onto the Snake Shed and Compound. 12 cores to the Diner, 6 cores each to the shed and compound. Single mode glass fibre is used throughout however we may start using Multi-Mode within buildings.

Terminating fiber isn't that difficult these days. Basically you just need cutters and strippers, cleaning wicks, guillotine and fusion splicer. We hire the fusion splicer and guillotine and purchase pre-made tails with ST connectors from AFC. Ends up being around $50 an end instead of the usual $100 an end for a contractor.

Pics:

Home wireless

Today I finally set up my little public access wireless network at home. I've had a Mikrotik RB133 with RB52 and Senao radios laying around in an outdoor box for sometime now. Today I fixed up the roof cable run and installed it on my existing mast with Cisco 21dBi solid dish.

I used a homebrew 8dBi collinear omni-directional antenna for the public side (Senao 200mW/23dBm) radio - that puts output at around 30dBi in a 360°H/25°V area. I've set up a rudimentary hotspot service on the interface - I will set it up with PayPal etc shortly so the neighbors can gain ad-hoc internet access and not feel guilty for it. Not everyone wants to pay for a permanent connection unbelievable huh.

I attached the dish to the RB52 radio which is adjustable - so I'll set that down to give no more than the regulated 36dBi. I'm hoping to establish a Point-to-Point link to a friend in Buderim. Failing that maybe a repeater link via Kiel Mountain which sits in between. Mainly to share links and to muck around with routing etc.

I'll go for a cruise around the local area and see how much coverage my little 8dBi is providing - might even be able to pick it up at Kiel Mt with the laptop radio...

Progress

Wireless Network

How things move slower the lager the enterprise. Might have to start pushing to get things done faster and more efficiently in future.

I installed the last two Access Points in from the first batch of six we received all those months ago. I guess a fairly large hold up was the flaky firmware of the Symbol WS5100 I mentioned earlier. It's going very nicely now and I can't wait to start tweaking it and including it into a radius/ldap domain.

One AP was installed in the main warehouse and another under the Taj for coverage of the main hallways. Both work quite well using the supplied 6dBi collinear antennas.

Today a 9dBi collinear from the RFShop arrived with two made up cables to suit the AP300. Another three access points will be ordered shortly to provide coverage of the Foodcourt - using the new 9dBi, open area between the warehouse and Taj - using existing 10dBi yagi and indoors are at the Croc's Lair shop - AP with integrated antennas. That will be the coverage are for now with the possibility of a few more sites later on.

Telstra IP WAN

Almost 6mths in the making and the completion date is within sight. The last week I have sorted out the issue Mooloolaba Travel were having with running their VPN to Galileo running over the new Telstra WAN. It simply wasn't connecting. Discovered that the 1-to-1 NAT addressing that was also used for IPSec and PPTP traffic did not like traffic coming from the direction of Mooloolaba. So they changed it to a Many-to-1 NAT and things are looking good apart from frequent dropouts of the IPSec VPN which I believe are due to the 'keepalives' being blocked - so I turned them off and I will see what happens.

WhaleOne still hasn't been changed over. I had a 3hr crack at it today with all the information I had scrounged from Telstra and the 'net in general. Turns out that their NextG account hasn't yet been set up for use with the IP WAN. What happens is that the APN Name changes from 'telstra.internet' to 'telstra.corp' and this dictates the gateway used for the connection. Before this can occur the SIM card/Account needs to be set up so that the calls are allowed through or something along those lines anyway - the request goes to the Telstra Mobile team. Hopefully it will be fixed up by Tuesday next week.

Myth Frontend

Ages ago I talked about building a mythbox out of a Epia M10000. Well last weekend I finally did it. Epia support is quite good with linux these days, it was easy to set up Mythbuntu on the box and have it connect to the existing Soltek Qbic 3401 Mythbox. It runs okay apart from stutter after a channel change for about 5 minutes - I think more RAM will fix that.

I might upgrade my PC and use the old bits to build a better backend someday.

Shoveling Data

Because the 'zoo keeps many operations such as Graphic Design and Marketing in-house it generates a considerable amount of data on a day-to-day basis. It's a challenge to keep all this centralised and backed up. What I have done to achieve ample storage with basic redundancy is use a 'front end' NAS (Network Attached Storage) combined with a 'back end' NAS located elsewhere from the front end serving as the primary backup/archive.

The users access the front end NAS directly and generally work from its shares. This will change in future as I intend to access it as a iSCSI mount on a server. This NAS is a standard box housing JBOD and runs OpenFiler - you have probably read about it here earlier. The performance of this NAS is fairly ordinary but since its the network that presents the bottleneck its not something to be concerned about at this stage.

The back end NAS is a purpose built NAS from a company called Thecus, the N5200. It houses up to 5 SATA disks and supports RAIDs 0 thru to 10. I've set up this particular one with 5 x 750GB disks with RAID5. This provides enough space to backup the front end NAS at maximum capacity - about 2.5TB total.

I'm currently backing up the front end NAS via rsync to the Thecus. I had to find the rsync 'module' to install on the Thecus first as it doesn't support it by default however it wasn't a difficult process.

I will consider a Thecus 1U4500 NAS to go with a future Novell OES2 server - mounting it as an iSCSI volume for localised e-mail/data archiving. This will probably use another 5200 for backups.

Overall this provides us with a sizable storage pool at a very reasonable cost. I would like to implement a proper SAN however our needs aren't that great at this stage and a single form of redundancy appears to be acceptable to management. I'll always plan for the upgrade though.

OpenDNS Media Release

Thought I'd link this here:

Australia Zoo Conserves Bandwidth, Enjoys 100 Percent Network Uptime with OpenDNS

Now we're getting somewhere

It has been a hectic few weeks for me. Major projects for the period include the preparations for Steve Irwin Day, the roll out of the new Telstra Next IP managed WAN and the slowly but surely deployment of a campus wireless network.

I'm also feeding in various nifty services into the Zoo network - such as a OpenFire Jabber server, Twiki wiki, One or Zero Helpdesk and a few other things. Although I'm going to hold off on their deployment to users until I have a LDAP directory of some sort in place for all these things to authenticate against.

I updated the firmware on my Sony Ericsson P1i too - the difference in performance and stability is night and day - and it was pretty good to begin with! The Opera browser and unified messaging apps have been improved quite a bit. I'm actually encoding TopGear episodes to 3gp format on my MythTV box and watching them during my lunch breaks on it - I didn't think I'd be using it like that. nb: I could just watch the xvid/dvix encoded eps but they're a tad large...

Steve Irwin Day
Steve Irwin Day went well as far as the web servers went - they handled a doubling of traffic without a hitch. I will be sad to see the replication servers go, they've done their job well and I'm kinda proud of them. I'm starting an upgrade of massive proportions of the two main web servers this week - hopefully once I'm done they'll be more than capable of handling the load without the need for replicas. More on that later.

Telstra NextIP
I've cut everything over to the zoo's shiny new Telstra Next IP WAN (to use their marketing spin). Speeds are good, response times are awesome. I'm also using Telstra's Proxy Caches too as they're very snappy and are used by many - plus there's a discount on data used through them apparently.

As part of the WAN, each SHDSL connected site has a managed Cisco 1801 router and SHDSL TA, some yumcha device. The NextG connection is as per usual, but when it is connected it has a L2TP into the WAN and thus has access to all the same routes as the other sites. I've set up Mikrotik routers at each site including the NextG connection - the Zoo has 2 x Yawarra WRAP1-2s in a rack enclosure still, Mooloolaba has a Yawarra WRAP1-1 with wireless and WhaleOne has a Mikrotik RB133 in an indoor enclosure.

Campus Wireless
Not too much progress - the Admissions indoors area now as its own AP and the Taj (crocosium buidling) offices have coverage too. I'm waiting for a sparky to run new cabling to key points so I can locate a few more APs in good coverage areas. Also waiting on a $1k order for various bits and pieces from the RFShop so I can start making up tails and prepare the splitters for installation. Also getting some antennas from them that are cheap and appear to have excellent performance - looking forward to trying them out.

Symbol/Motorola WS5100

If you use these Wireless switches and are still running pre-3.0 firmware, UPDATE! Huge changes made and I suspect it's Motorola weaving its magic. I had all sorts of issues running with Spectralink VoWiFi sets and coverage - updated firmware to 3.0.2.0 and everything is happy now.

Other benefits of the firmware is that the CLI now mimics Cisco's IOS in many ways and the Java/Web interface is greatly improved - information is readily available and the controls make sense...

It's changed my view on this kit I was almost about to turf it in exchange for some Cisco gear or even Mikrotik (but I didn't really want to configure each AP individually).

Posting from a P1i

Well, it'll take some getting used to but it isn't that hard.

Planning for high traffic

The most worrying project on my list at this time is working out how to achieve as much grunt as possible to withstand the estimated traffic from the first Steve Irwin day tribute website since his passing.

Currently the Zoo has a single primary server hosting all sites and a secondary web server sharing the load on a few sites. It's not the perfect model by far and I intend on tidying it up as follows.

What I plan to do is install MySQL 5.x on the secondary server and ready it for replication. Then I will dump the contents from the existing MySQL 4.x databases and point all the websites at it. Once I'm satisfied that it's functioning as expected (I'm in the process of testing this on the bench). Then I will upgrade the MySQL 4.x to 5.x on the primary server and begin multi-master replication with the other. This completes stage one of the preparations.

With the databases in place and functioning, I will work out how to replicate all vhosts between the two servers. Plesk, the web control panel operating on both servers, makes this more complex than it really needs to be since I will also need to create the client/domain accounts for each of the replicated sites. When I have worked out what is required I will script the synchronization as much as possible and hopefully end up with near 100% automation. I may need to tap into Plesk's API to do this. This will complete stage two.

With replication of both the databases and general structures taking place between the two existing servers I can now introduce more servers and the greater complexity they will bring.

I will be working towards four application servers just for static content/scripts and a single localised database server. I intend to just have raw boxes running either Redhat or FreeBSD, no fancy control panels getting in the way. This will allow me to script everything with simplicity and provide a basic configuration to each server. I will replicate the data from the first of the existing servers to one of the new application servers and from there to each of the remaining three. This is so I can keep the amount of public traffic between the servers to a minimum. I will introduce the new database server into the multi-master replication loop and point the four new application servers at it. This completes stage three.

Once I am satisfied that each application server is connecting to the database server over their private network and that the database server is successfully replicating the databases from the existing two servers I will set up the load balancer to include the four new application servers. We may need to cut over to a new load balancer since the existing one may not support this many servers.

This setup will provide me with six front end servers and three database servers - with a bit of sharing of resources here and there. The following diagram shows what I intend on achieving.



Relevant links:
ONLamp Advanced MySQL Replication Techniques
MySQL 5.0 Manual - Replication
Rsync
SWSoft Plesk - Upgrading MySQL

Attending to the web servers

Web servers are like the wilder beasts of the Internet. I imagine them to be out in the open grass plains happily grazing in the sun. I also can imagine tigers, cheetahs and other predators lurking around the fringes looking for the few that aren't paying attention or have been wounded and thus are falling behind the herd.

The predators are the numerous script kiddies (skiddies) and crackers out there that either trying it on, seeking to find yet another server to host their warez, or they're building a massive, high bandwidth botnet in which to strike down those who oppose them.

When I looked at my new herd of servers (okay, 3 isn't really a herd...) I saw a neglected bunch that needed some tender loving care. So all this week I have been looking at what makes them tick and for what purposes they serve. In the process I've been cutting the fat, optimizing and tightening things up. There's still a ways to go but things are already looking better, especially after upgrading the link from 10Mbit to 100. I hope whoever owned those domains I disabled doesn't get too pissed.

Things that need to happen:

• Consolidate servers into a single rack and connect them via a private LAN
• Adjust load balancing to go between all three servers
• Establish a managed firewall
• Upgrade OS and packages on each

I won't feel comfortable until all that is in place.

Amazing what a bit of tweaking does

The big long stretch is where the poor MySQL daemon just couldn't cope with the demand. So a few tweaks to how it caches etc later and you see the boost. DB Server load dropped from 45.0 down to 0.40 too.

The joys of e-mail administration

I've dabbled in e-mail services for sometime now. It's one of those things that would normally be within the sysadmin's domain but usually falls on the netadmins task list. I think it's something to do with the diagnostic/trouble shooting process - it's pretty much the same as most network issues.

This week was 'fix the mail server' week. Resurrect it and get the mail flowing as it should.

The mail server is a moderately new 'oem' box with average kit and runs CentOS 4. For some reason sendmail is what they have used, personally I have always liked postfix - especially when its combined with policyd.

Sendmail was having a lot of trouble sending e-mail to a few domains that were also fairly popular among the users. I quickly narrowed the problem down to a flaky link causing connections to time out - likely a issue with using PPPoE over wireless and then going though some magical shaping gateway to the 'net. So I set up forwarding to the service providers IronPort mail server - once I had figured out the particulars of getting sendmail to forward via an authenticating mta, the outgoing mail queue was kept nice and empty.

Once that was done, I then focused on the viral aspect of email. It appears the anti-virus in use on the mail server was way out of date and while its defs were up to date, the engine simply couldn't detect many of the popular worms. So I left it as it was and installed ClamAV - it's doing the job fine.

Next was figuring out how they were using procmail to process messages. This is where I discovered, to my displeasure, that they were using procmail to run spamassassin and the anti-virus, along with some basic procmail type spam filtering. What a waste of resources processing mail at the mailbox stage is. So I've shifted those tasks to the MTA where they belong. Procmail is for users to distribute mail among folders and vacation messaging when .forward isn't enough.

So now the server has basic virus and spam filtering abilities once again. Next step is to look at shifting over to postfix and implementing various policy daemons with their grey/white/black listing, SPF, spamtraps, HELO checking and weighted scoring goodness. I will also use amavis-new or xamime to run stuff through a few anti-virus scanners and deal with mail accordingly.

With all these changes I was forced to implement a rather draconian policy of limiting message sizes to 10MB. This was all that the ISPs mail server would accept, not that I disagree - it's e-mail, not FTP... So being the friendly BOFH I had to offer my flock an alternative to send those large files to outside recipients. In comes PaknPost, a http upload/emailer webapp. It's written in perl and free - what more could I ask for? This allows users to send up to ten files to anyone they like, with virus scanning, file encryption and HTTPS transfer. I'm quite happy with the initial results, user abuse will be the ultimate test.

Setting up for Production

Things are swinging into production now. Two sites have gone live in 'Test Mode' - once I'm happy all the bugs are gone I'll push them into production and see how it all goes. Here's hoping.

I started to play around with the FreeRADIUS rlm_perl module to see about modifying radius requests/replies before they hit the database. Main reason is to swap those bloody Colubris accounting values. I must be getting good at this Perl stuff because it was stupidly easy. Now I have a single perl script sitting in radius land that shuffles data around - so many possibilities have opened up with this level of control.

Curtin is still a ongoing interest. Agreements have been signed and supposedly the project should be in full swing. And so it should be, they have specified a end of April deadline. However I haven't even received equipment, nor has it even been ordered. Something about leasing it or something - I don't care because if they want me to finish the preparation before I leave then I'll need to see something next week at the latest.

Hotspot Website Details

As promised, more details on the Hotspot interface. This will make it a lot easier to roll your own anyway.