September 15, 2014
Integrating Spirent into an Automated Workflow Test Methodology

This blog post is from: 
I’ve spent the last few days getting briefed by several vendors in Silicon Valley.  They include A10, Big Switch, Brocade, Cisco, Gigamon, Nuage, Pluribus, Spirent, and Thousand Eyes.  Over the next few weeks, I’ll try and get a few posts out about the briefings, but for the first one I wanted to focus on Spirent.  Many are probably aware that Spirent provides packet generators and while that’s what they sell and are really good at, it’s the strategy, vision, and software integration with their products that was extremely intriguing.  

I’ve engaged with many customers over the past 10 years and the majority have never felt a real need to test performance.  It was and is usually very easy to over provision hardware when it comes to Layer 2 & 3 switching.  This is still the case for the most part too – there are 1 RU and 2 RU switches that can forward traffic faster than those big monster boxes of just a few years ago.

Why Test Now?

There are network functions being virtualized from almost every vendor out there —- this usually falls under the Network Functions Virtualization (NFV) trend.  It’s common to hear about virtual routers, load balancers, firewalls, and other services appliances, but it is still uncommon to see them in deployments.  Why?  Fear, uncertainty, and doubt (FUD) and vendor limited performance/features have a lot to do with it.  How will the virtual firewall perform?  How well does it match the data sheet numbers?  Can a virtual load balancer handle the SSL offload required for a particular environment?  What happens if more memory or CPU is added to the virtual appliance – does it help or hurt?

That all said, software appliances as well as hardware appliances aren’t going anywhere.  They will each have their place in the network, but for the 1000s of small to mid-size environments out there, there is no reason that virtual appliances couldn’t be used.  And for physical appliances, do you really want to over-provision?  As bandwidth speeds are hitting high density 10G, 40G, and 100G on switching platforms, this is NOT the case on L4-7 appliances like FW & LB.  If you try and match those speeds on L4-7 devices, you’ll need to spend millions of dollars.  This is another reason testing totally makes sense nowadays.

How do you validate the L4-7 appliances?

Clearly, Spirent can be used to validate the performance of the appliances.  If you want to deploy a virtual FW/LB, you can probably even use the 30 or 60 day trial version of that product to test with.  It is worth noting Spirent has virtual appliances that can be used to test traffic rates up to 5Gps.  Last time I checked, 5Gbps is still greater than most Internet circuits, thus these appliances can be used to potentially test load balancers and firewalls that will reside on the perimeter or within the data center of most Enterprises.

It’s the process, time, and effort that usually slows things down though.  To do this (before the virtual Spirent appliances came out) in the past, you would have to rent/lease/buy physical appliances, get them racked and cabled, learn all about Spirent, and hope you are performing the right test.  Let’s say all that happens to go smooth and shortly after, you make a change to your firewall and you no longer have the Spirent appliances.  Do you re-validate the performance after the change (and assume the change won’t affect performance) or was the initial perf test just about seeing what the max throughput rates were on the appliance?  While a one-time test is good, what could make it even better?

In reality, the best case scenario is an integrated workflow that includes an engineer pre-building config changes, having the changes be reviewed by others on the team and other cross-functional teams as needed, have the changes deployed in a pre-prod (or prod) environment, and following the changes, have unit tests performed that include both functional and performance tests.  It’s this type of vision that Spirent shares, which is pure awesomeness.  It’s rare to see any networking company advocate for or have a Continuous Integration (CI) strategy integrating with tools like GitHub, Gerrit, or Jenkins.

Spirent is not the norm here and are definitely showing thought leadership.  Spirent has developed a Jenkins plug-in that will help to build the integration tests that need to happen after a given change, or code drop, to a git repository like GitHub.  You don’t need Git integration to use what Spirent has developed, but it helps to give a good example for a complete workflow.

Note: I actually planned to do a short Continuous Integration (CI) post before this one, but that didn’t happen.  Expect that to come soon and that may give more clarity on what’s possible with GitHub + Jenkins integration.

What does this mean?

You would build out the workflow and not have to spend countless hours performing the tests! Jenkins would kick off all required tests.  Tests could include anything you’d like to verify/validate configs and features, etc., but using the Spirent Jenkins plug-in, it would communicate to the Spirent device(s) to perform the required performance tests.  Again, this eliminates the need for a person to be running 10s and 100s of manual tests, which is just huge.  As CI methodologies become more popular in the network space, we’ll be able to know exactly what to expect in terms of features and performance after each and every change, while ensuring the changes are successful and don’t break anything.  Ideally, this gets done on the prod gear during the change window, but to get things started, there is no reason this can’t be done on lab kit especially if you’re testing virtual appliances!

It’s also worth noting Spirent has a virtual test center that customers can use as well to test particular configurations.  For example, you can test Open vSwitch, VMware VSS/DVS, or whatever you’d like. If it’s software like a FW/LB, just upload and kickoff an automated test on their hardware.  If you want to test on YOUR particular hardware, which is more in line with what I’d recommend, you can still use their Jenkins plugin to assist in building out a proper test methodology.

All of the sessions were recorded this week - feel free to check out the Spirent videos.

If you are interested in testing out virtual appliances, let me know.  This is an area I may be launching a service offering around in the near future based on customer feedback.


Twitter: @jedelman8

Disclaimer: Spirent was a presenter at Networking Field Day 8.  They did not ask for any consideration in the writing of this review nor were they promised any review.  The conclusions and analysis contained in this post are mine and mine alone.


Jason - Nice write up… 

Here is a quick blog post overview introducing what we covered: 

@Spirent can orchestrate physical and logical network topology mapping with MRV Media Cross Connect - 

History of @Spirent - we go all the way back to 1936! 

here is a link to the iTest Jenkins PlugIn - An agile enabled platform for network test automation.

Let us know how we can help!

September 4, 2014
History of bits and bytes and word sizes

In the beginning there was a bit.  A bit can be either on or off.  
Then we made bytes.  Bytes are typically 8 bits.
Even though we count in bytes the computers we use are still doing everything a word at a time.  The word size is based on the number of bits they can use for each operation.
The earliest computers and video game systems could work on one byte at a time and were 8 bit word sizes. Most computers used today are 64 bit.
Here is an idea of the addressing limits for the various word sizes. 
  • 8 bits = 2^8 = 256
  • 16 bit = 2^16 = 65,536
  • 24 bits = 16,777,215 bytes (2^24-1 bytes) 16 MB - 65535 times larger than 8 bit
  • 31 bit = 2 gigabytes 2,147,483,647 (2 GB) 128 times larger than 16 MB
  • 32 bit = 2^32 = 4,294,967,296 (4 GB)
  • 64 bit = 2^64 = 16 exabytes (EB) 18,446,744,073,709,551,616 = 4 billion times larger than 32 bit  = 16 million gigabytes.
An exabyte is slightly more than one billion gigabytes.
A gigabyte is 1,073,741,824 (10243 or 230) bytes.

August 29, 2014
high end mini x86 system

Gigabyte GB-BXI7-4500 Intel HM87 C i7-4500U 1.8GHz(16GB Max Memory) Intel HD Graphics, USB3.0, mSATA

MZ-MTE1T0BW-Samsung 1TB 840 EVO Series mSATA3 SSD 540 MBps (read) / 520 MBps (write) MZ-MTE1T0BW

Gigabyte GB-BXI7-4500 Intel HM87 C i7-4500U 1.8GHz(16GB Max Memory) Intel HD Graphics, USB3.0, mSATAUnit Price$499.95Item#SYSGIGBX451RMfg. Part#


Samsung 1TB 840 EVO Series mSATA3 SSD 540 MBps (read) / 520 MBps (write) MZ-MTE1T0BWUnit Price$469.95Item#DRISAM1TB03RMfg. Part#


August 17, 2014
Using Cryptographic Hashes to verify file download integrity

Using Cryptographic Hashes to verify file download integrity

NOTE: Don’t use MD5 (see below)

The SHA hash functions are a set of cryptographic hash functions designed by the National Security Agency (NSA) and published by the NIST as a U.S. Federal Information Processing Standard. SHA stands for Secure Hash Algorithm.

Vendors provide a sha-1 hash for software downloads. This enables you to verify that your downloaded files are unaltered from the original.

To confirm file integrity, use an sha-1 utility on your computer to calculate your own hash for files downloaded from the VMware web site.

If your calculated hash matches the message digest we provide, you are assured that the file was downloaded intact.

sha-1 utilities are available for Windows and Linux and Mac. Most UNIX installations provide a sha1sum command for sha-1 hashes. You may need a newer linux kernel to calculate the checksums for larger files.


The File Checksum Integrity Verifier (FCIV) can be used on Windows based products to verify sha-1 values.

See: for details on FCIV.

> fciv -sha1 test-file.exe

// File Checksum Integrity Verifier version 2.05.

26b3fa5790c6b1bfca80b9e67402ee3622253a71 test-file.exe

Mac OS X: How to Verify a SHA-1 Digest

Instructions on checking an sha-1 checksum on a Mac:
In Finder, browse to /Applications/Utilities.
Double-click on the Terminal icon. A Terminal window will appear.
In the Terminal window, type: “openssl sha1 ” (sha1 followed by a space).
Drag the downloaded file from the Finder into the Terminal window.
Click in the Terminal window, press the Return key, and compare the checksum displayed to the screen to the one on the vendor’s download page.

From TechNet

Windows Server 2008 R2 Standard, Enterprise, Datacenter, and Web (x64) – DVD (English)
File Name: en_windows_server_2008_r2_standard_enterprise_datacenter_web_x64_dvd_x15-50365.iso
Size: 2,858 (MB)
Date Published (UTC): 8/31/2009 10:22:24 AM
Last Updated (UTC): 1/11/2010 4:31:40 PM
SHA1: A548D6743129F2A02C907D2758773A1F6BB1BCD7
 ISO/CRC: 8F94460B

About MD5

MD5 was designed by Ron Rivest in 1991 to replace an earlier hash function, MD4. In 1996, a flaw was found with the design of MD5. While it was not a clearly fatal weakness, cryptographers began recommending the use of other algorithms, such as SHA-1 (which has since been found also to be vulnerable). In 2004, more serious flaws were discovered, making further use of the algorithm for security purposes questionable; specifically, a group of researchers described how to create a pair of files that share the same MD5 checksum. Further advances were made in breaking MD5 in 2005, 2006, and 2007. In an attack on MD5 published in December 2008, a group of researchers used this technique to fake SSL certificate validity.

US-CERT says MD5 “should be considered cryptographically broken and unsuitable for further use,”and most U.S. government applications now require the SHA-2 family of hash functions.


August 10, 2014
Everyday Special Chocolate Chip Waffle Recipe from Scratch IbenIT


3.5 cups flour

6 tsp baking powder

1 tsp salt

4 beaten egg yolks

3.5 cups milk

1 cup oil

4 stiffly beaten egg whites

1 cup chocolate chips

Sift dry ingredients together in 10 cup mixing bowl. This bowl will

be used to dispense the batter to the waffler maker.

Combine yolks, milk, and oil in an 8 cup mixing bowl. Stir wet into


Fold in chocolate chips.

Fold whites leaving a few fluffs.

Bake 3/4 cup at a time. Makes 8 6” round belgian style waffles.

August 4, 2014
Things Hercule Poirot says. 

There are still a few unaired episodes that are coming out soon. 

Poirot says: “I cannot eat these eggs. They are of totally different sizes!”

Things Hercule Poirot says.

There are still a few unaired episodes that are coming out soon.

Poirot says: “I cannot eat these eggs. They are of totally different sizes!”

August 3, 2014

It’s amazing! DisplayPort to the new Asus PB287Q 4k display is truly 4 times the resolution of the old 1080p HMDI.

I just added my 3rd 1080HD screen so now there are 6 times the standard desktop space of just 1 screen.

Get yours now - in stock at your local Central Computer store. 

Plugs right into the displayport on your PC. Works at 3840 by 2160 and 30 Hz with the AMD Radeon 5800 series card and I’m still able to drive total of 3 displays - DP, HDMI, and DVI.

Resolution Calculator Spreadsheet

Acer X233H192010801.782,073,60012,073,600$62.69$130$130Asus VH232H192010801.782,073,60012,073,600$62.69$130$130Asus PB287Q384021601.788,294,40018,294,400$78.37$650$650sub total768043205.3312,441,600312,441,600$67.92$910$910—3840x2160-uhd-4k-monitor-1ms-gtg—pb287q-monasupb287r.htm

  • Asus PB287Q 28” 3840x2160 UHD 4K Monitor 1ms GTG
  • 60Hz Refresh Rate
  • Price $649.95
  • Item#MONASUPB287R
  • Mfg. Part#PB287Q

July 12, 2014
Don Cheadle wears a Kangol Flat Cap Hat in the movie volcano

Don Cheadle wears a Kangol Flat Cap Hat in the movie volcano

July 4, 2014
Using Red Hat RDO PackStack to build #OpenStack #Neutron

Many people using OpenStack will start off with Devstack running in a VM on a laptop to get a feel for what is possible but when running VMs or playing around with networking and storage options and running nova to different hypervisors is needed it’s time to install a “real” openstack distribution.  You can build openstack from scratch using the manual installation steps which takes hours but is an excellent learning experience or you can use one of the assisted installation distribution methods.

Red Hat is the leader in the Enterprise Linux space. They make RHEL, CentOS, and Fedora and output code which is used by many other distributions.

Suse, Debian, Canonical Ubuntu and other distros are gaining in popularity and striving to differentiate themselves in the market.

Packstack is a CLI tool created by RedHat employee that uses puppet to script the installation and configuration of OpenStack components on a redhat based operating system.

RedHat started RDO with community-supported packages of the most up-to-date OpenStack releases available for download. They also offer enterprise-level support with the Red Hat Enterprise Linux OpenStack Platform.

To get started with RedHat RDO and install OpenStack in 15 minutes you need a machine with a running OS already.  Open a root shell and a web browser and head over here and follow the 3 simple steps to get setup.

This will get you up and running quickly.  What is great is that an answer file is created which can be edited and reused as more sophisticated configurations are needed. So it’s a great place to start but also offers many sophisticated options under the hood.

Here’s a list of questions you might like to explore:

Welcome to Installer setup utility
Enter the path to your ssh Public key to install on servers [/root/.ssh/id_rsa. pub] :
Should Packstack install MySQL DB [y|n] [y] :
Should Packstack install OpenStack Image Service (Glance) [y|n] [y] :
Should Packstack install OpenStack Block Storage (Cinder) service [y|n] [y] :
Should Packstack install OpenStack Compute (Nova) service [y|n] [y] :
Should Packstack install OpenStack Networking (Neutron) service [y|n] [y] :
Should Packstack install OpenStack Dashboard (Horizon) [y|n] [y] :
Should Packstack install OpenStack Object Storage (Swift) [y|n] [y] :
Should Packstack install OpenStack Metering (Ceilometer) [y|n] [y] :
Should Packstack install OpenStack Orchestration (Heat) [y|n] [n] :
Should Packstack install OpenStack client tools [y|n] [y] :
Enter a comma separated list of NTP server(s). Leave plain if Packstack should n ot install ntpd on instances.:
Should Packstack install Nagios to monitor OpenStack hosts [y|n] [y] :
Enter a comma separated list of server(s) to be excluded. Leave plain if you don ‘t need to exclude any server.:
Do you want to run OpenStack services in debug mode [y|n] [n] :
Enter the IP address of the controller host [192.168.x.x] :
Enter list of IP addresses on which to install compute service [192.168.x.x] :
Enter list of IP addresses on which to install network service [192.168.x.x] :
Do you want to use VMware vCenter as hypervisor and datastore [y|n] [n] : 
Enter the IP address of the VMware vCenter server to use with Nova: 192.168.x.x
Enter the username to authenticate on VMware vCenter server: 
Enter the password to authenticate on VMware vCenter server :
Enter the name of the vCenter datastore: wLSI
Enter the IP address of the MySQL server [192.168.x.x] :
Enter the password for the MySQL admin user :
Confirm password :
Set the AMQP service backend [qpid|rabbitmq] [rabbitmq] :
Enter the IP address of the AMQP service [192.168.x.x] :
Enable SSL for the AMQP service? [y|n] [n] :
Enable Authentication for the AMQP service? [y|n] [n] :
Enter the password for the Keystone admin user :
Confirm password :
Enter the password for the Keystone demo user :
Confirm password :
Enter the Cinder backend to be configured [lvm|gluster|nfs|vmdk] [lvm] : 
Enter the CPU overcommitment ratio. Set to 1.0 to disable CPU overcommitment [1 6.0] : 
Enter the RAM overcommitment ratio. Set to 1.0 to disable RAM overcommitment [1 .5] : 
Enter the bridge the Neutron L3 agent will use for external traffic, or ‘provide r’ if using provider networks [br-ex] :
Enter the name of the L2 plugin to be used with Neutron [linuxbridge|openvswitch |ml2] [ml2] :
Should Packstack install Neutron LBaaS [y|n] [n] : y
Should Packstack install Neutron L3 Metering agent [y|n] [n] : y
Would you like to configure neutron FWaaS? [y|n] [n] : y
Enter a comma separated list of network type driver entrypoints [local|flat|vlan|gre|vxlan] [vxlan] :
Enter a comma separated ordered list of network_types to allocate as tenant networks [local|vlan|gre|vxlan] [vxlan] :
Enter a comma separated ordered list of networking mechanism driver entrypoints [logger|test|linuxbridge|openvswitch|hyperv|ncs|arista|cisco_nexus |l2population] [openvswitch] :
Enter a comma separated list of physical_network names with which flat networks can be created [*] :
Enter a comma separated list of physical_network names usable for VLAN: 
Enter a comma separated list of <tun_min>:<tun_max> tuples enumerating ranges of GRE tunnel IDs that are available for tenant network allocation: 
Enter a multicast group for VXLAN: 
Enter a comma separated list of <vni_min>:<vni_max> tuples enumerating ranges of VXLAN VNI IDs that are available for tenant network allocation [ 10:100] :
Enter the name of the L2 agent to be used with Neutron [linuxbridge|openvswitch] [openvswitch] :
Enter a comma separated list of bridge mappings for the Neutron openvswitch plugin: 
Enter a comma separated list of OVS bridge:interface pairs for the Neutron openvswitch plugin:
Enter interface with IP to override the default tunnel local_ip: 
Enter VXLAN UDP port number [4789] :
Would you like to set up Horizon communication over https [y|n] [n] : y
Enter the path to a PEM encoded certificate to be used on the https server, leave blank if one should be generated, this certificate should not require a passphrase:
Enter the SSL keyfile corresponding to the certificate if one was entered:
Enter the CA cahin file corresponding to the certificate if one was entered:
Enter the Swift Storage devices e.g. /path/to/dev:
Enter the number of swift storage zones, MUST be no bigger than the number of storage devices configured [1] :
Enter the number of swift storage replicas, MUST be no bigger than the number of storage zones configured [1] :
Enter FileSystem type for storage nodes [xfs|ext4] [ext4] :
Enter the size of the storage device (eg. 2G, 2000M, 2000000K) [2G] :
Would you like to provision for demo usage and testing [y|n] [y] :
Would you like to configure Tempest (OpenStack test suite). Note that provisioning is only supported for all-in-one installations. [y|n] [n] : y
Enter the network address for the floating IP subnet [] :
What is the uri of the Tempest git repository? [] :
What revision, branch, or tag of the Tempest git repository should be used [master] :
Enter the IP address of the MongoDB server [192.168.x.x] :
Enter the password for the nagiosadmin user :
To subscribe each server to EPEL enter “y” [y|n] [y] :
Enter a comma separated list of URLs to any additional yum repositories to install:
To subscribe each server to Red Hat enter a username :
To subscribe each server to Red Hat enter your password :
To subscribe each server with RHN Satellite enter RHN Satellite server URL:

June 30, 2014
Russian government to switch from Intel and AMD x86 “WinTel” solutions to Linux running on ARM CPU

Is this a good idea or just security through obscurity?

Russian government to switch from Intel and AMD x86 “WinTel” solutions to Linux running on ARM CPUs. 8 core in 2015 and 16 core for servers in 2016.

Liked posts on Tumblr: More liked posts »