Network upgrade

And the upgrades continue.

The network here is expanding something chronic so I needed something that could push the vlans harder. It's basically replacing the local Mikrotik/WRAP1-1 Router and Asus GigaX2024 L2 switch with a single Asus GigaX3112 L3 switch. It certainly tidied up the rack by removing two switches and a stack of patch leads.

Now I have two Asus 2024 L2 switches to stick in Admissions and the Crocosium. This will give me a gigabit vlan trunk to the locations and allow me to create some more subnets to reduce some of this needless traffic off the main networks. I'm trying to have a L2 managed switch on the end of every fibre link to get some flexibility into the network and get things into this decade...

I'm not all that impressed with the Asus network kit so far. It's okay for the price but it's buggy as hell and the 3112 tends to crash due to kernel panics and reboot due to buffer overflows or memory errors. I'm hoping future firmware updates will come and fix things. Not that I had much more luck with Netgear and Linksys stuff. There's a reason why Cisco can charge so much.

The campus wireless network is almost completed. Awaiting a cable run from the Machinery shed to the Conference center to install the AP on the roof there. Also need to install the 12th AP at the Tiger Temple to complete the 'ring of coverage' - full wireless coverage of the safari shuttle track and nearby walkways and buildings. Will put off any more expansion until the Hotel is built.

An A/V Perspective

Here's a picture of the A/V crew operating the two remote PTZ cameras and 'house' audio gear during the Veronica's gig at Australia Zoo.

The panel between them is the Panasonic Visual Mixing desk. The smaller monitors are the cameras and the screen on the top is what's currently showing on the big screen (a rather large Panasonic LED screen outside).

Progress

Wireless Network

How things move slower the lager the enterprise. Might have to start pushing to get things done faster and more efficiently in future.

I installed the last two Access Points in from the first batch of six we received all those months ago. I guess a fairly large hold up was the flaky firmware of the Symbol WS5100 I mentioned earlier. It's going very nicely now and I can't wait to start tweaking it and including it into a radius/ldap domain.

One AP was installed in the main warehouse and another under the Taj for coverage of the main hallways. Both work quite well using the supplied 6dBi collinear antennas.

Today a 9dBi collinear from the RFShop arrived with two made up cables to suit the AP300. Another three access points will be ordered shortly to provide coverage of the Foodcourt - using the new 9dBi, open area between the warehouse and Taj - using existing 10dBi yagi and indoors are at the Croc's Lair shop - AP with integrated antennas. That will be the coverage are for now with the possibility of a few more sites later on.

Telstra IP WAN

Almost 6mths in the making and the completion date is within sight. The last week I have sorted out the issue Mooloolaba Travel were having with running their VPN to Galileo running over the new Telstra WAN. It simply wasn't connecting. Discovered that the 1-to-1 NAT addressing that was also used for IPSec and PPTP traffic did not like traffic coming from the direction of Mooloolaba. So they changed it to a Many-to-1 NAT and things are looking good apart from frequent dropouts of the IPSec VPN which I believe are due to the 'keepalives' being blocked - so I turned them off and I will see what happens.

WhaleOne still hasn't been changed over. I had a 3hr crack at it today with all the information I had scrounged from Telstra and the 'net in general. Turns out that their NextG account hasn't yet been set up for use with the IP WAN. What happens is that the APN Name changes from 'telstra.internet' to 'telstra.corp' and this dictates the gateway used for the connection. Before this can occur the SIM card/Account needs to be set up so that the calls are allowed through or something along those lines anyway - the request goes to the Telstra Mobile team. Hopefully it will be fixed up by Tuesday next week.

Myth Frontend

Ages ago I talked about building a mythbox out of a Epia M10000. Well last weekend I finally did it. Epia support is quite good with linux these days, it was easy to set up Mythbuntu on the box and have it connect to the existing Soltek Qbic 3401 Mythbox. It runs okay apart from stutter after a channel change for about 5 minutes - I think more RAM will fix that.

I might upgrade my PC and use the old bits to build a better backend someday.

Shoveling Data

Because the 'zoo keeps many operations such as Graphic Design and Marketing in-house it generates a considerable amount of data on a day-to-day basis. It's a challenge to keep all this centralised and backed up. What I have done to achieve ample storage with basic redundancy is use a 'front end' NAS (Network Attached Storage) combined with a 'back end' NAS located elsewhere from the front end serving as the primary backup/archive.

The users access the front end NAS directly and generally work from its shares. This will change in future as I intend to access it as a iSCSI mount on a server. This NAS is a standard box housing JBOD and runs OpenFiler - you have probably read about it here earlier. The performance of this NAS is fairly ordinary but since its the network that presents the bottleneck its not something to be concerned about at this stage.

The back end NAS is a purpose built NAS from a company called Thecus, the N5200. It houses up to 5 SATA disks and supports RAIDs 0 thru to 10. I've set up this particular one with 5 x 750GB disks with RAID5. This provides enough space to backup the front end NAS at maximum capacity - about 2.5TB total.

I'm currently backing up the front end NAS via rsync to the Thecus. I had to find the rsync 'module' to install on the Thecus first as it doesn't support it by default however it wasn't a difficult process.

I will consider a Thecus 1U4500 NAS to go with a future Novell OES2 server - mounting it as an iSCSI volume for localised e-mail/data archiving. This will probably use another 5200 for backups.

Overall this provides us with a sizable storage pool at a very reasonable cost. I would like to implement a proper SAN however our needs aren't that great at this stage and a single form of redundancy appears to be acceptable to management. I'll always plan for the upgrade though.

OpenDNS Media Release

Thought I'd link this here:

Australia Zoo Conserves Bandwidth, Enjoys 100 Percent Network Uptime with OpenDNS

Now we're getting somewhere

It has been a hectic few weeks for me. Major projects for the period include the preparations for Steve Irwin Day, the roll out of the new Telstra Next IP managed WAN and the slowly but surely deployment of a campus wireless network.

I'm also feeding in various nifty services into the Zoo network - such as a OpenFire Jabber server, Twiki wiki, One or Zero Helpdesk and a few other things. Although I'm going to hold off on their deployment to users until I have a LDAP directory of some sort in place for all these things to authenticate against.

I updated the firmware on my Sony Ericsson P1i too - the difference in performance and stability is night and day - and it was pretty good to begin with! The Opera browser and unified messaging apps have been improved quite a bit. I'm actually encoding TopGear episodes to 3gp format on my MythTV box and watching them during my lunch breaks on it - I didn't think I'd be using it like that. nb: I could just watch the xvid/dvix encoded eps but they're a tad large...

Steve Irwin Day
Steve Irwin Day went well as far as the web servers went - they handled a doubling of traffic without a hitch. I will be sad to see the replication servers go, they've done their job well and I'm kinda proud of them. I'm starting an upgrade of massive proportions of the two main web servers this week - hopefully once I'm done they'll be more than capable of handling the load without the need for replicas. More on that later.

Telstra NextIP
I've cut everything over to the zoo's shiny new Telstra Next IP WAN (to use their marketing spin). Speeds are good, response times are awesome. I'm also using Telstra's Proxy Caches too as they're very snappy and are used by many - plus there's a discount on data used through them apparently.

As part of the WAN, each SHDSL connected site has a managed Cisco 1801 router and SHDSL TA, some yumcha device. The NextG connection is as per usual, but when it is connected it has a L2TP into the WAN and thus has access to all the same routes as the other sites. I've set up Mikrotik routers at each site including the NextG connection - the Zoo has 2 x Yawarra WRAP1-2s in a rack enclosure still, Mooloolaba has a Yawarra WRAP1-1 with wireless and WhaleOne has a Mikrotik RB133 in an indoor enclosure.

Campus Wireless
Not too much progress - the Admissions indoors area now as its own AP and the Taj (crocosium buidling) offices have coverage too. I'm waiting for a sparky to run new cabling to key points so I can locate a few more APs in good coverage areas. Also waiting on a $1k order for various bits and pieces from the RFShop so I can start making up tails and prepare the splitters for installation. Also getting some antennas from them that are cheap and appear to have excellent performance - looking forward to trying them out.

Symbol/Motorola WS5100

If you use these Wireless switches and are still running pre-3.0 firmware, UPDATE! Huge changes made and I suspect it's Motorola weaving its magic. I had all sorts of issues running with Spectralink VoWiFi sets and coverage - updated firmware to 3.0.2.0 and everything is happy now.

Other benefits of the firmware is that the CLI now mimics Cisco's IOS in many ways and the Java/Web interface is greatly improved - information is readily available and the controls make sense...

It's changed my view on this kit I was almost about to turf it in exchange for some Cisco gear or even Mikrotik (but I didn't really want to configure each AP individually).

Planning for high traffic

The most worrying project on my list at this time is working out how to achieve as much grunt as possible to withstand the estimated traffic from the first Steve Irwin day tribute website since his passing.

Currently the Zoo has a single primary server hosting all sites and a secondary web server sharing the load on a few sites. It's not the perfect model by far and I intend on tidying it up as follows.

What I plan to do is install MySQL 5.x on the secondary server and ready it for replication. Then I will dump the contents from the existing MySQL 4.x databases and point all the websites at it. Once I'm satisfied that it's functioning as expected (I'm in the process of testing this on the bench). Then I will upgrade the MySQL 4.x to 5.x on the primary server and begin multi-master replication with the other. This completes stage one of the preparations.

With the databases in place and functioning, I will work out how to replicate all vhosts between the two servers. Plesk, the web control panel operating on both servers, makes this more complex than it really needs to be since I will also need to create the client/domain accounts for each of the replicated sites. When I have worked out what is required I will script the synchronization as much as possible and hopefully end up with near 100% automation. I may need to tap into Plesk's API to do this. This will complete stage two.

With replication of both the databases and general structures taking place between the two existing servers I can now introduce more servers and the greater complexity they will bring.

I will be working towards four application servers just for static content/scripts and a single localised database server. I intend to just have raw boxes running either Redhat or FreeBSD, no fancy control panels getting in the way. This will allow me to script everything with simplicity and provide a basic configuration to each server. I will replicate the data from the first of the existing servers to one of the new application servers and from there to each of the remaining three. This is so I can keep the amount of public traffic between the servers to a minimum. I will introduce the new database server into the multi-master replication loop and point the four new application servers at it. This completes stage three.

Once I am satisfied that each application server is connecting to the database server over their private network and that the database server is successfully replicating the databases from the existing two servers I will set up the load balancer to include the four new application servers. We may need to cut over to a new load balancer since the existing one may not support this many servers.

This setup will provide me with six front end servers and three database servers - with a bit of sharing of resources here and there. The following diagram shows what I intend on achieving.



Relevant links:
ONLamp Advanced MySQL Replication Techniques
MySQL 5.0 Manual - Replication
Rsync
SWSoft Plesk - Upgrading MySQL

What have I been doing?

Been a while since I last posted so time for a update post!

Revamping the old network
The network at the Zoo was a miserable mess - it took me a while to audit what was in place and to devise a topology that would best address the current and future needs of the Zoo. Now the Zoo has a VLAN'd network consisting of dedicated Administration, Point-of-Service and VoIP subnets, OSPF routing at the core, a DMZ, traffic policing and shaping capabilities and VPN (PPTP/L2TP and IPSec) capabilities.

I achieved all this by using two rackmounted WRAP1-2's from Yawarra and a cheap Asus GigaX 2024 switch. I loaded Mikrotik RouterOS 2.9.42 onto the two WRAP1-2's and set up 802.1q VLANs on the switch. The VLANs are routed on the first WRAP1-2 which then connects onto the DMZ where the other WRAP1-2 and Cisco 857/877 routers exist with OSPF routing throughout. The second WRAP1-2 holds up the 1Mbit Unisky wireless connection (PPPoE, over wireless... yuk) and hopefully a substantial fibre based service from someone, such as a 2Mbit E1/G.703 service.



I used pairs of Cisco 8xx series routers to hold up VPN links between the Zoo and its newly opened Mooloolaba retail store. A pair of 857's hold up a general Point-of-Service/LAN traffic IPSec tunnel. In addition to that a pair of 877's hold up a VoIP/Video IPSec tunnel with QoS. The two DSLs are 8Mbit/384Kbit links supplied by Bigpond. Having dedicated 'pairs' of routers/DSL for VPN connectivity is overkill but it's still cheaper than a single fibre service.

These changes provided the framework for the following additions to the network infrastructure.

A new phone system
The Zoo's old Siemens key system was well and truely past its time and was needing upgrades which proved to be exorbitantly costly to do. A new Alcatel OmniPCX PBX was selected and installed by company called Nexon Asia Pacific. Along with the digital and analog extensions a number of VoIP extensions are provided including wireless VoIP sets. Best practice says to establish a dedicated subnet for the PBX/VoIP services to reside within so as to isolate it from the general traffic of the other networks. Having VLAN capability is useful as I can locate the phones nearly anywhere and still keep them within the VoIP subnet. However while the phones support VLANs, they don't want to communicate with the Asus switch.

First it was wireless and whales, now its wireless and... um... elephants?
I will soon have a Zoo wide wireless network built up of Symbol WS5100 and AP300s. These were provided by Barcode Dynamics in addition to inventory/asset tracking equipment. The WS5100 is useful in that it can map VLANs to WLANs - allowing me to simply create wireless extensions of the existing networks with no physical modifications. However security becomes a concern with the absence of a router/firewall - the WS5100 addresses this by supporting WPA1/2, 802.1x and firewall policies. I will also limit transit between the networks and wireless infrastructure via the routers.

To start with the wireless will be used for mobile VoIP. Since the Alcatel mobile sets are basically Spectralink reference designs I can simply apply the pre-configured Spectralink QoS policy on the WS5100 to that WLAN so that it grants expedited access to the wireless bandwidth to VoIP traffic. In the future we will also implement mobile Point-of-Service terminals, either PDA style units or small form factor PCs. There's also the possibility that the roaming photographers could also use the coverage to upload their digital photo's in real-time to the on-site photography lab.

I've just finished setting up a outdoor enclosure for one of the AP300's. It's a pity that the AP300 doesn't have an outdoor variant. The supplied enclosures were just bare boxes, luckily they came with the backing board. However I had to make up the pole brackets myself using some angle brackets, u-bolts and pop-rivets.





Mobile VPN over Telstra's NextG
For the newly launched Whale One vessel the Zoo has established a NextG mobile data service. To connect the boat to this service a ruggedised NextG modem/router was installed on the boat with a 7dBi collinear antenna. The router comes with a PPTP VPN client so I have set this to establish a VPN back to the Zoo. This allows the two Point-of-Service terminals to communicate back to the Zoo's POS services for EFT transactions and accounting/stock control. Under testing we managed to maintain a connection out to 10km to sea and sustain an average data rate of 1.5Mbit/sec. I have yet to test the link with the POS systems running.

Attending to the web servers

Web servers are like the wilder beasts of the Internet. I imagine them to be out in the open grass plains happily grazing in the sun. I also can imagine tigers, cheetahs and other predators lurking around the fringes looking for the few that aren't paying attention or have been wounded and thus are falling behind the herd.

The predators are the numerous script kiddies (skiddies) and crackers out there that either trying it on, seeking to find yet another server to host their warez, or they're building a massive, high bandwidth botnet in which to strike down those who oppose them.

When I looked at my new herd of servers (okay, 3 isn't really a herd...) I saw a neglected bunch that needed some tender loving care. So all this week I have been looking at what makes them tick and for what purposes they serve. In the process I've been cutting the fat, optimizing and tightening things up. There's still a ways to go but things are already looking better, especially after upgrading the link from 10Mbit to 100. I hope whoever owned those domains I disabled doesn't get too pissed.

Things that need to happen:

• Consolidate servers into a single rack and connect them via a private LAN
• Adjust load balancing to go between all three servers
• Establish a managed firewall
• Upgrade OS and packages on each

I won't feel comfortable until all that is in place.

Amazing what a bit of tweaking does

The big long stretch is where the poor MySQL daemon just couldn't cope with the demand. So a few tweaks to how it caches etc later and you see the boost. DB Server load dropped from 45.0 down to 0.40 too.

The joys of e-mail administration

I've dabbled in e-mail services for sometime now. It's one of those things that would normally be within the sysadmin's domain but usually falls on the netadmins task list. I think it's something to do with the diagnostic/trouble shooting process - it's pretty much the same as most network issues.

This week was 'fix the mail server' week. Resurrect it and get the mail flowing as it should.

The mail server is a moderately new 'oem' box with average kit and runs CentOS 4. For some reason sendmail is what they have used, personally I have always liked postfix - especially when its combined with policyd.

Sendmail was having a lot of trouble sending e-mail to a few domains that were also fairly popular among the users. I quickly narrowed the problem down to a flaky link causing connections to time out - likely a issue with using PPPoE over wireless and then going though some magical shaping gateway to the 'net. So I set up forwarding to the service providers IronPort mail server - once I had figured out the particulars of getting sendmail to forward via an authenticating mta, the outgoing mail queue was kept nice and empty.

Once that was done, I then focused on the viral aspect of email. It appears the anti-virus in use on the mail server was way out of date and while its defs were up to date, the engine simply couldn't detect many of the popular worms. So I left it as it was and installed ClamAV - it's doing the job fine.

Next was figuring out how they were using procmail to process messages. This is where I discovered, to my displeasure, that they were using procmail to run spamassassin and the anti-virus, along with some basic procmail type spam filtering. What a waste of resources processing mail at the mailbox stage is. So I've shifted those tasks to the MTA where they belong. Procmail is for users to distribute mail among folders and vacation messaging when .forward isn't enough.

So now the server has basic virus and spam filtering abilities once again. Next step is to look at shifting over to postfix and implementing various policy daemons with their grey/white/black listing, SPF, spamtraps, HELO checking and weighted scoring goodness. I will also use amavis-new or xamime to run stuff through a few anti-virus scanners and deal with mail accordingly.

With all these changes I was forced to implement a rather draconian policy of limiting message sizes to 10MB. This was all that the ISPs mail server would accept, not that I disagree - it's e-mail, not FTP... So being the friendly BOFH I had to offer my flock an alternative to send those large files to outside recipients. In comes PaknPost, a http upload/emailer webapp. It's written in perl and free - what more could I ask for? This allows users to send up to ten files to anyone they like, with virus scanning, file encryption and HTTPS transfer. I'm quite happy with the initial results, user abuse will be the ultimate test.

Another intense week

Today I spent the morning at the new Australia Zoo "On the Beach" shop opening. Basically making sure everything IT wise went smoothly, and that it did, until the afternoon when the main link into the Zoo decided to drop causing them to fail... that was an interesting hour.

I'm endeavoring to maintain the level of client satisfaction that I desire in the given environment. There needs to be changes made to streamline desktop support as much as possible to allow IT to concentrate on how to improve on other services such as telephony and core services. Plus there needs to be time given to proper planning and implementation with the necessary change control procedures. I guess I'm asking to be allowed to take a proactive approach to IT services.

The highlight of the week was seeing a wombat riding in a trolly/cart type thing.

First week at the 'zoo

It's been very intense. Point-of-sales terminal upgrade across the board, working alone on Saturday and planning and configuring various highly technical functions in a extremely short period of time. That's just a few of the many tasks I faced during the first week on the job.

There is much that needs to be done however the time frames for doing so is worrying. For example setting up dual DSL connections for load-balancing with VPNs between Mooloolaba and the 'zoo in a matter of hours isn't something I would like to do often. Other future plans are to upgrade the PBX system incorporating VoIP, site wide Wireless coverage and E-mail services upgrades.

There is also a LOT of tidying up to do of existing services. I will be working on various scenarios on how to address the zoo's requirements while also trying to reduce vulnerabilities, effort and cost.

Another note - I need to brush up on my 'controlling client expectations' exercises.

The end of one saga, the beginning of another

Today marks a turning point in my IT career. I have officially finalized my employment at AccessPlus and tomorrow I will continue my career at the Australia Zoo.

The send off wasn't that extravagant, a simple lunch with Don and Andrew and at the end of the day I said my goodbyes and left without further discussion.

I will continue to have something to do with Marinanet - that's still yet to be decided. I will also continue to consult independently to local businesses and individuals on their wireless/network, linux/bsd, OSS needs. Be it on a purely part time basis.

Given appropriate authority I will continue writing about my work at the 'zoo. I feel it will prove just as interesting, hopefully more, as my work at AccessPlus.

Induction day at the Zoo

Yesterday was pretty much my first day working at Australia Zoo. The whole time was dedicated to introducing us to the operational side of the zoo, what to do, what not to do. It was fairly intense with a lot of information to absorb in one day but it was very useful knowledge none-the-less.

The parts I found most interesting were the health and safety, privacy and security aspects. I haven't been exposed to an organisation that was so much within the public eye before. So simple things like "don't point out visiting celebrities" I would never had considered.

The staff there seem like a happy crowd that will be good to work with. Plus there's a lot of variety in roles and personalities so there won't be any of that small business monotony to contend with. However, being a reasonably large organisation, it will be my best interest to stay away from any gossip/rumors that tend to breed in such environments.

A change in career

Today I gave notice at AccessPlus. I have accepted a position as systems administrator at the Australia Zoo beginning the 5th of April.

My primary role will be administrating two web servers located in a datacentre in the US and looking after the local e-mail services. I will also assist in desktop support.

Still yet to put detail into my initial plans for the new job but I have some lofty goals in mind for the current environment. I'm hoping my responsibilities will expand into the netadmin side of things, it's only natural for me to pursue my comfort zone no?

As a result of this move I have lost my webhosting capability at AccessPlus. Thus I have set up a Google Applications Account and shifted my domain over - so now www/mail/blog/docs/start.naturalnetworks.net all point to one google app or another. I'll be using this as a kind of fancy wiki - documenting my knowledge and publishing articles. Still need a place to store downloads though.