Reading and remembering

As I was reading scripture today, I came upon Matthew 8, particularly the section that describes what one has to be willing to surrender to follow Jesus. I have read this dozens of times, but I had to "cheat" using Bible Gateway to remember where it was in scripture.

Something that I know already, but want to try more today, is sharing something you read and remember with other people. That is the best way to learn anything, not just where something is in scripture. I'm learning a new language right now, and it's well-known that the way to retain what you learn in a day is to repeat it and interact with it among lots of people.

May I be better at walking that out...

Application Performance II

I thought I would just tell a few quick stories about my encounters with application efficiency and how the network was blamed. In the spirit of humility, I'll offer where I feel I did poorly and where I did well, and what lessons I learned.

City Police Database
The biggest, most obvious example of this was when I was doing some consulting work for a city government, which shall of course remain nameless. They had gotten a new application (and when I say new, I mean it--it had not been seriously proven in any other installations). I was asked to make the application high-availability. The application was not ready to be installed yet, but I was to do the up-front work. The parameters were thus:

• We don't want to buy any new network equipment
• The redundancy must go across multiple sites, which are connected by DS3 links several hops away.
  

So I devised a method I thought was pretty good to fit these parameters. It used pre-existing Cisco 3845 routers: one at the "main site" and one at the "backup site." The 3 servers that made up the program (two application servers and one database server) went on a new VLAN that was routed by the 3845. The VLAN at the main site had one particular IP subnet and the VLAN at the backup site had another subnet. But each server had a loopback interface, and those interfaces were the ones that DNS pointed to. The 3845 at the main site used Cisco's IP SLA with tracking, tracking the application's TCP port on it's physical address. If the application went down, it stopped advertising the loopback /32 via EIGRP. The backup site would then start advertising, because it had a route to that /32 address but with an administrative distance of 200, which is lower than EIGRP's external administrative distance of 170.

All that is background, which is actually not related to my point. All of this redundancy stuff should probably not have been put in, at least without a careful up-front study of the application itself. That's another lesson: do your homework first, and save a lot of headache. If you have built a Cadillac simply to transport monkeys, you've probably wasted your time and efforts (and the customer's money). My personality type is ISTP, which if you know psychology, tells you why I went ahead and did all this.

It turns out that the application was horribly written in many ways--it didn't run as a service, it ran as a user-level program, so we had to set the servers to auto-login, etc. But the biggest problem was, and the point of this section, the performance. It had never been load-tested, nor had it been run over anything less than a gigabit network. So when it was slow, it was of course the fault of the network, and more specifically, the network engineer who obviously didn't know what he was doing.

Here's what we determined after I did a careful analysis of the application (without seeing any code or finding this out from the programmers who didn't speak English very well). The client component of the application, which sat on the user's workstation, was doing SQL queries in the background. Instead of doing a query like:

• select COMPONENT from TABLE where SOMETHING > 100;

it was doing something like:

• select * from TABLE;

then once it downloaded the huge amount of data, it would "post-filter" that. Yes, definitely a network problem when every query resulted in the downloading of over 100 megs of data, which would then be weeded out by the client software just to show 10 records.

It was really difficult maintaining humility in this, and I must confess that though I did maintain humility while investigating, I blew a gasket once I discovered the problem.

Pharmaceutical Database
There was a particular pharmaceutical company I consulted for who wanted a fast link between their main site and another site where many users were as well as off-site backups sat. They already had a DS3 between the sites, but they wanted to put in an additional 100 Mbps connection, which they did. I then configured all the routers, etc, to route things primarily down the 100 Mbps connection and secondarily down the DS3.

Everything was fine and we verified the routing was going the correct way, etc. But then the database backups started going much slower than they had before. After investigating, I discovered that the DS3 ran more-or-less directly from their office in West Bend, WI, to the office in Waukesha, WI. However, the 100 Mbps connection ran from West Bend, WI, up to the CO in St Paul, MN, then down to Waukesha, WI.

Obviously, the latency over the 100 Mbps connection was significantly more (25-30 milliseconds) than the latency over the DS3 since the distance was so much further. This can be understood here. So this was a network problem, if you will. However, again, the way the database was synchronizing, it was doing thousands of little queries, rather than a bulk transfer.

Here were some possible solutions, in no particular order.

1. Modify all the computers to use a higher TCP window size using the Window scale option. Without window scaling, you have to consider latency when determining the maximum speed of a TCP session. It goes like this:
   
   Window Size (bytes)
   ------------------- * 8 bits/byte = bps
   Latency (sec)
   
   So 65535 / .025 * 8 = ~21 Mbps. So we could not get more than 21 Mbps from this connection.
   
2. Make the syncing go over the DS3.
3. Ask the programmers re-write the way the database synchronizes.
4. Make the carrier reroute the 100 Mbps connection.
   

First we enabled TCP window scaling option for the two servers that needed to synchronize across this connection. That didn't seem to help. Then we dug into just how the application was working and discovered the little queries that made the syncing work, which would be unaffected by window sizing. So we tried option 3, because it was what we determined to be the best. The programmers laughed and said it was a networking problem and it was our fault because it used to work better. We probably should have pushed them harder and gotten management involved more, but we didn't. We tried option 4 but that was impossible because the carrier was based in MN and would have charged roughly a bazillion dollars to run fiber directly between the two sites, and this carrier was originally chosen precisely because they did a good price/sales job on the management.

Ultimately we made the syncing go over the DS3 using offset lists in EIGRP for /32 routes for those two servers. That also meant that there would be asynchronous routing when anything communicated across the WAN with one of those two servers (besides the server-to-server communication of course). No one cared, as long as it "worked better" and redundancy was there.

Two lessons from these experiences:

• Do your homework! Don't rush ahead without asking a lot of good questions. Some things may seem obvious based on your past experience, but don't assume.
• If you determine something to be the best option and it gets laughed off by the people you're suggesting it to, go to management ready to explain all the ins-and-outs of the problem as you understand it.
• Whatever your personality type, attempt to show restraint in running ahead with solutions. Brain-storm, but remember that brainstorming is not meant to be implemented on a whim. Sleep on it and run it by colleagues and other smart people.
  

Remember, work as a team to get it resolved. If you blame someone else, try to put yourself in their shoes and always speak with humility and remind people you are on the same team--you just want to find the best solution. This kind of thinking is often contagious, and usually management will appreciate it. Of course, this isn't always true, and you can't control other people, but if you keep your cool, things will go better, at least for you.

Application Performance and the Jesus Way

Application Efficiency, Part I
The website Ethereal Mind, which I often read, had a recent post in response to another post by Matthew Norwood entitled Programming Bad Performance. It touched on something every network engineer feels when application performance is slow somewhere on the network. It's a topic I have dealt with a lot in my professional career so I uncharacteristically weighed-in with a comment, which I present below in a modified form...

It seems that the onus, or bur­den of proof, is often on the net­work engin­eer to fig­ure out just what the prob­lem is, since the net­work is this mys­ter­i­ous entity that most people don’t know much about. I have found two types of server admins in my exper­i­ence:

• The ones that assume it’s their prob­lem and never think about the net­work, but then it proves to be a net­work prob­lem (often when attempt­ing to estab­lish com­mu­nic­a­tion between a ser­vice net­work and an internal net­work).
• The ones that assume (along with all the users, usu­ally) that it is a net­work prob­lem and very quickly start point­ing fin­gers.

I have found that over the years I have learned a lot more about effi­cient data­base pro­gram­ming, ana­lyz­ing server cpu, memory, and disk util­iz­a­tion, etc, than I ever wanted to, as a means to simply find what the real prob­lem is. Greg at Etherealmind points out that prov­ing it’s not the net­work, or at least cast­ing plaus­ible reason to believe it’s not, will get you home on time. I agree with that, but in a world of people say­ing, “It’s not my prob­lem” and wash­ing their hands of it, I want to put forth a little more effort to get to the bot­tom of things.

Something else that I have learned is that humil­ity is really import­ant. If you start act­ing cocky and the prob­lem turns out to be yours, you really look like, and are, a jerk. Taking more of a “Let’s fig­ure this out together” atti­tude is much more likely to lead to suc­cess and team unity in the end. The real art is being able to foster and main­tain that atti­tude when other people seem to be out for blood.

The Jesus Connection
I'll make this my first post that crosses my two categories of blogging. Wanting to live life in a way that Jesus would do it, the previous paragraph helps see one or two ways of doing that. The bible often speaks of humility... three places that jump out to me:

• Do nothing out of selfish ambition or vain conceit. Rather, in humility value others above yourselves, not looking to your own interests but each of you to the interests of the others (Philippians 2:3-4).
• For those who exalt themselves will be humbled, and those who humble themselves will be exalted (Matthew 23:12).
• When pride comes, then comes disgrace, but with humility comes wisdom (Proverbs 11:2)

Unity is also key to solving problems. It's amazing how humility and unity go together as well (I think that's pretty obvious). If everyone is cocky, it does not help team unity--it just creates divides. Be completely humble and gentle; be patient, bearing with one another in love. Make every effort to keep the unity of the Spirit through the bond of peace. There is one body and one Spirit, just as you were called to one hope (Ephesians 4:2-4).

Three of my favorite topics: humility, unity, and hope!

Go out with joy

As I was praying this morning, and studying the scriptures, I remembered that going out with joy is one of the most powerful testimonies to the love of God. Some people just seem to have joy and a hope, even if circumstances are crummy.

It reminds me of the famous passage in the Prophet Isaiah chapter 55 (which I wrote some about last year). Verse 12 says,

• "You will go out in joy and be led forth in peace;
  
• the mountains and hills will burst into song before you,
  
• and all the trees of the field will clap their hands."

How is it to go out in the joy of the Lord and be so contagious that the trees of the fields and the mountains will even catch the joy? Obviously it's poetic, but what better way to speak of the fellowship with God than to display joy and hope as you go along?

Where I live, everyone around me says, "As-Salamu Alaykum" which basically means "Peace be upon you" in Arabic--a beautiful greeting. I also long for a joy to be among us all through knowing the very character and nature of God and how he extends his hand in love and calls us each by name (John 10:3)!

As-Salamu Alaykum!

Psalm 27

It's amazing how the same section of scripture that you have read many times comes to life anew every time.

The Lord is my light and my salvation, whom shall I fear?

This is a basic rule for life. Even if someone kills me or I die, because the Lord is my eternal salvation, where is there room for fear? I'm not my own salvation--I haven't saved myself through my piety or good deeds.

The LORD is the stronghold of my life—of whom shall I be afraid?

If the last sentence speaks of my eternal salvation, this one speaks of my life here and now. Fear is one of the most devious tricks of Satan. It paralyzes a person from carrying out the things they know they should do.

One thing I ask of the LORD,
this is what I seek:
that I may dwell in the house of the LORD
all the days of my life,
to gaze upon the beauty of the LORD
and to seek him in his temple.

The most important thing, the goal of life, the highest request to be made of God, is to dwell in his presence, to gaze on his beauty, to bask in his love.

For in the day of trouble
he will keep me safe in his dwelling;
he will hide me in the shelter of his tabernacle
and set me high upon a rock.

God is faithful. If we look to him and remain in his presence, he will reach back to us and do these things.

at his tabernacle will I sacrifice with shouts of joy;
I will sing and make music to the LORD.

The natural response for the deliverance that is offered.

More on this another day, but I'm trying to keep these short. There is a lot to fear in India. Going out on the streets trying to practice language is really difficult for me, whether it's because I'm not particularly extroverted, or because if I look silly, I want it to be on my own terms, or some other reason. Yet the people I would interact with are not my enemies at all--most of them are happy to help me learn. Yet it's easier to just stay in the house.

Yet none of these fears should be serious. Real fear is not having enough money to buy food, or not having a secure place to stay. Yet these "big" fears like that are easier to tackle head-on and to give a name to. It's the small fears that constantly bombard a person. Yet God has promised release from fear, and that a life lived in fear is not a life of obedience.

I'm writing these things to remind myself... :)

The Language of [Jesus]

Since living in India, it's interesting to see the different religious terminologies, based on different languages, traditions, and religions. I've discovered that most (if not all) major religions in India have a reverence for Jesus Christ. But there is confusion about his name, and his title, perhaps.

• The Name
• The name he probably went by in his time was "Yeshua"
  
• "Jesus" that we use in English seems to have come primarily from the Latin, "Iesous."
• Most Hindi speakers will say "Yeshu"
  
• He is known in the Arabic from the Qur'an as عيسى ("Isa") and that has become the name all Muslims will use, Arabic-speaking or not.
  
• Whatever language, the meaning is the same; the name means "God is salvation."
  

• The Title
• The original title is, of course, Hebrew: Mashiaẖ
  
• When this title is translated into Greek, it is "Christ"
  
• In English we say "Messiah" or "Christ"
• In Arabic, as defined in the Qur'an, his title is "al-Masīḥ" and that has become the title all Muslims will use, Arabic-speaking or not.
• Whatever language, the meaning is the same; his title means "Anointed One."

Home Router and DD-WRT

I just discovered DD-WRT , which is replacement firmware supporting a large number of SOHO routers (e.g., Netgear, Linksys, D-Link, etc). I tried it on my new D-Link DIR-615 and so far it seems to be working well. One of the big things I was going for was QoS (Quality of Service) which can prioritize things based on some criteria. For example, I prioritized Skype so my wife and I can have good conversations with people back in the US, even if someone else is transferring a large file.

There are a lot of other slick features as well, for example, there are instructions on how to create a tunnel with popular brokers, such as SixXS and Hurricane Electric.

Since DD-WRT is Linux, it opens up a world of possibilities as well, for example, ssh access combined with Dynamic DNS allows remote logging in for testing purposes, etc.

IPv6 with Cisco ASA, Windows 2008, Linux

It's my first blog post in nearly a year. One of my new years resolutions is to post at least three times/week. We'll see how it goes.

Recently I was implementing IPv6 using the devices mentioned above. The Cisco ASA firewall code was 8.2(1), which supposedly supports IPv6... it seems unless you have a failover pair. If you set a static IPv6 address, the IPv6 on the interfaces will shut themselves down because of a DAD (duplicate address detection) failure as both the active and the standby firewalls will assume whatever address you put in. With IPv4, you can enter a "standby" address on each interface so that won't happen. The way around it on a failover pair is to use the EIU-64 address (the prefix+the 48-bit MAC address and padding between the vendor part and the device part). Therefore, both firewalls will have unique addresses and there will be no conflict. The problem with this is:

• The addresses become full sized monsters
• The address will change in the event of the standby firewall going active
  
• Both firewalls give out RA (router advertisement) messages

With Linux, if I hard-code a default route, it will pay attention to that, primarily. If I hard-code a default route in Windows 2008, it seems to simply add it to the other ones it obtained via the RA messages it received. As a result, IPv6 will be quite flaky on Windows with two routers, if one is not forwarding traffic (in the ASA world, the standby firewall will not route anything).

I can think of two solutions:

• Upgrade the ASA to 8.3 code, which allows IPv6 failover commands and should work fine. The problem with this is that you will have to upgrade the RAM, which is a pain.
• Suppress the RA messages from the ASA. This can be done with the " ipv6 nd suppress-ra" interface configuration command.

Obviously the first solution is better, but the second one works if upgrading is not possible.

Cheers!
Matt