Dropped PING

We recently had a situation where two servers on the same subnet would periodically loose contact with each other.  After going through the common troubleshooting steps:

• Verify IP / Subnet mask / Gateway
• Make sure that NIC drivers were correct
• Verify that the server port channels / interface configurations were correct...

We still had issues.  We after reviewing the arp cache on one of the servers, we were seeing two different MAC addresses.  Periodically, the correct server MAC would be populated, and periodically a different MAC would be show.

Ultimately, the issue was resolved after we discovered that a Cisco ASA on the same subnet had proxy arp enabled.  This was causing the arp cache of the server to periodically get populated with the wrong MAC (mac of the ASA vs the MAC of the target server.)

Disallow SCCM during the day

We kept having issues with Forfront updates being pushed out to clients during the day.  The updates would get pushed immediately to all workstations (before the site distribution points got the update.)  Unfortunately, 10-20 clients taking a modest update at the same time can still crush a T1 or other small WAN circuit.

The solution / workaround for us was to completely block communication to our SCCM system during normal production hours (ex: 7:00 AM - 6:00 PM) at the sites.

The rule looks slightly different depending on the type of hardware / circuit.  For example, a Cisco router with a T1 would look similar to this:

time-range WorkWeek
periodic weekdays 7:00 to 18:00
exit

ip access-list extended DenySCCM
deny   ip host 1.1.1.1 any time-range WorkWeek
permit ip any any
exit

interface serial0/0/0
 ip access-group DenySCCM in    
exit

While a Cisco ASA might look more like this:

time-range WorkWeek
periodic weekdays 7:00 to 18:00
exit
access-list DenySCCM extended deny ip 1.1.1.1 255.255.255.255 any time-range WorkWeek
access-list DenySCCM extended permit ip any any
!
Int VLAN2
access-group DenySCCM in interface inside
!

Network Police

The following examples show how to limit excess traffic to a set of "servers" to 500 Kbps and limit bandwidth to a particular "site" to 2 Mbps.  This is useful for throttling traffic on WAN interfaces.

1) Define a the servers and site

object-group network servers
 host 1.1.1.1
 host 1.1.1.2
 host 1.1.1.3
 host 1.1.1.4
!
object-group network site
 2.2.2.0 255.255.255.0
!

2) Define the class maps (will correllate to the access-lists in step 3)

class-map match-any site_traffic
 match access-group 199
 match  precedence 3
class-map match-any server_traffic
 match access-group 198
 match  precedence 3
!

3) Define an access-list that matches both the "access-group" in step 2 and the "object-group" in step 1.

access-list 198 permit ip any object-group servers
access-list 199 permit ip any object-group site
!

4) Define the limits that you want to apply.

policy-map QoS
 class servers
    police 500000
 class site
   police cir 2000000
!

5) Apply the policy to the interface that you wish to control.

interface FastEthernet0/1
 service-policy output QoS

Without these rules the network traffic could easily exceed the bandwidth available.  For example without the rules, attempted traffic across a 10 Mbps circuit for the site and servers might look like this.

Once the rules are applied, netflow allows us to see the containment of the traffic:

CSV File Cleanup

If you ever wanted to save a google search to a CSV file, you should check out SEOquake.

Recently I used SEOquake to generate 50 CSV files.  The end goal was to work with these CSV files in Powershell.  Unfortunately, each of the original files had a header that I wanted to remove.

I used the following powershell code to strip out the first 6 lines of each of these files:

After running that powershell code, I can now work with SEOquake output more easily inside of other powershell scripts.

Data Center Cleanup: Part 2

The next step in the cleanup process was deploying switches to each rack, commonly referred to as Top of Rack (ToR).  We opted for a slightly different twist on this, and decided to install the switches in the bottom of the rack.

With ToR, when a cable is too long, there is the potential for droop.

With this in mind, we opted to install the switch at the bottom of the rack.  This way, excess cable collects at the bottom of the rack.  

Still a bit messy, but another step toward forward.

Data Center Cleanup: Part 1

For quite some time, we had discussed "cleaning up" our data center.  Past installations had made ongoing support tricky.  For example, it was easy to accidentally bump a power cord and unplug a server.  Additionally, it was very difficult to see / trace cables.

Fortunately, as part of a new project, we were given the green light to address the situation.

Here is an example of two of the server racks in our data center (before):

Clearly there were many issues that needed to be addressed.  Where to start?

Most of the original power cords that came with the servers were 8 - 10 feet long black power cords.

Step 1: Color code and shorten the power cords lengths.  
We swapped the original cords out with 2' red and blue power cords.   The color coding now allows us to quickly confirm that each server (dual power supply server) is running on both the A & B PDU.  Additionally, the shorter cables have allowed us to start to eliminate the extreme amount of excess cabling in our racks.

Here are the power cords that we removed from just 1 cabinet.  The excessive length of the power cords had obstructed our view of the servers.  Additionally, the weight of the cables made it VERY easy for the cords fall out of the PDU.

Although this is just a beginning, at the end of the first step (left cabinet), we could finally start to glimpse the servers (after step 1).

Nested loops

A number of years ago, the company I worked for needed to generate a large set of unique codes. The codes needed to be 5 digits long, and we wanted to remove the letters I and O and the numbers 1 and 0 from the listing.  In an effort to avoid letting my PowerShell skills rust, I decided to recreate this project in PowerShell.

As I started thinking about ways to approach this, it occurred to me that I rarely seem to need to use nested loops.  I assume there are other ways to solve this, but nesting seemed like fun.  Maybe this outline will prove useful for someone looking to learn a little bit more about nesting.

I broke the problem down into two parts:

1. Create an alphanumeric array that conforms to the requirements (removing the I,O,1, and 0)
2. Create a series of nested loops, with each loop representing 1 of the places in the 5 digit code

Once finished, the output was written to a .CSV file, and it looked similar to the small sample below:

BH3NA

BH3NB

BH3NC

BH3ND

BH3NE

BH3NF

BH3NG

If you were to let this run to completion, I believe it would generate 24,000,000 + unique codes (this post is about PowerShell, not Finite mathematics).