80.0 Network Design Cisco Forum FAQ80.0 Network Design
·Network Design Tips
·Running Out Of IP Addresses due to 'flat network' design
·Tips in Designing Network on Hub-and-Spoke, Full-Mesh, or Partially-Mesh setup
·Tips in connecting hosts to switches
·Resilient Network Tips for Small Businesses
·BGP Design
·Migrating network
Network Design Tips (#15009)
»Cisco Forum FAQ »I do not understand subnetting?
In network design, there are several basic considerations such as scalability and reliability. Reliability is about stable and seamless communication between hosts. Scalability is about how a network growth is anticipated properly with minimal change.
To have scalable and reliable network design, all Layer 1 to Layer 3 good side aspects should be met. Following are certain factors that affect network scalability and reliability, from Layer 1 to Layer 3.
Layer 1
1. Cables
Cable Management
Don't
* Don't run cables on floor
* Don't wrap network cables and power cords into one bundle
* Don't use too-long or too-short cables
Do
* Cables should run over (the ceiling), under (the tile), or inside (the wall)
* Have cable slacks between devices for easy and proper cable work space
Network Cable Choice
Use fiber cable around your building for best performance, scalability, and reliability. When you have multiple devices that employ various cable types (i.e. coax, Category 3, Category 5); you might want to have some kind of multiplexer to multiplex all of those various cable types into a single fiber cable.
When you do have to use Category 5 cables to interconnect devices, then it is suggested to use Category 6 instead of using Category 5 or 5E cables. Category 6 cables are more reliable and flexible to any network environment.
2. Racks
When you have multiple network devices that need to be in the same room, it is then suggested to have dedicated racks for them. In addition, the rack itself should also be mountable to the floor and/or to the wall for steady standing position.
The assumption is that the network devices should also be rack mountable. If the network devices are not rack mountable, the devices then should be wall or desk mountable.
3. Power
Between AC and DC Power
Most common network devices are probably AC-powered. By nature, AC-powered devices consume more energy and produce more heat than DC-powered devices. One big no-no on any network devices are having too-high temperature environment, which will shut down (or even melt down) the devices.
Whenever possible, use DC-powered network devices to reduce power bill. This is true especially when you need cooling system to cool down your network devices.
UPS (Uninterruptable Power Supply), Electrical Wiring, and Power Drop
Network devices are quite sensitive to "dirty power" that might affect resources such as CPU and memory chips. With UPS, the network devices would receive cleaner power and constant power supply with its battery backup.
Keep in mind that having UPS itself might not be sufficient. You may also need to verify your building electrical wiring and power drop from your Utility company.
As illustration, verify that the ground prong on your three-prong outlet is wired properly to the Earth. This way, you are sure you have proper power supply end to end.
Layer 2
Broadcast Storm
To support a lot of users, some people like to span a large Layer-2 network. The network is usually looking like at least one of the following:
* Employ three or more switches in daisy-chain connection
* Employ more Layer-2 switches rather than Layer-3 switches or routers
* One VLAN is spread throughout at least almost the entire organization
* One VLAN covers large area
* Use subnet size larger than /24 on one VLAN
* Assign multiple subnets under the same Layer-3 interface
As illustration, the network setup could look like the following
»Cisco Forum FAQ »Resilient Network Tips for Small Businesses
When the Layer-2 network is a broadcast network, then there will be something called ARP broadcast. This ARP broadcast is used to establish Layer-2 communication among hosts within the same Layer-2 network. As any other traffic, ARP broadcast traffic consume bandwidth and network device resources such as CPU and memory.
The more hosts reside within the Layer-2 network, then there will be more ARP broadcast traffic take place. The larger ARP broadcast traffic take place, the less bandwidth and network device resource available for the actual communication between two hosts within the same Layer-2 network. As a note, a large ARP broadcast traffic is sometime referred as broadcast storm.
One way to reduce broadcast storm effect is to partition a Layer-2 network into smaller multiple Layer-3 networks. Usually it is the best approach to use Layer-3 switches to partition such Layer-2 network since you can keep the Layer-2 switching speed eventhough you are using Layer-3 routing to route traffic among the Layer-3 networks.
The Layer-2 network partition should also consider native VLAN partition (usually VLAN 1). When there are multiple Layer-2 switches, you don't really need to let native VLAN (or any VLAN) to spread across the entire network. You could just segment VLAN 1 into smaller multiple VLAN 1 networks and use Layer-3 switching to interconnect them. This way, any behavior change on native VLAN (due to maintenance or DOS attack) would not bog down the entire network.
Layer 3
Subnet Assignment and Contiguous Network
Let's review the following network design
Internet
|
|
Firewall
|
|
Router 4
|
|
Switch
| | |
+----------------+ | +-------------------+
| | |
Router 1 Router 2 Router 3
| | | | | |
1st | 3rd | 4th 5th network
2nd network Switch
| | |
+---+---+--------------------+-+-+-+-+--------------------+---+---+
| | | | | | | | |
Server | Server Server | Server Printer | Printer
Server Server Printer
The network design represents the network setup of an entire organization network. Router 1 is managing three networks. Similarly Router 2 is managing one network and Router 3 is managing two networks. All of these networks are broadcast networks.
There is also another broadcast network within the four routers themselves. Network between Router 4 and Firewall is point-to-point network. Network between the Internet (ISP) and Firewall could be any network type (point-to-point, broadcast, or non-broadcast).
To provide connectivity within the organization, a 192.168.0.0/24 subnet is used. Let's say for now that this subnet must be sufficient to support the entire network within the organization.
Let's say we have the following host number to support within each network
1st network: 30 hosts
2nd network: 20 hosts
3rd network: 10 hosts
4th network: 5 hosts
5th network: 3 hosts
Server farm: 12 hosts
Each network would be independent network. There will be IP routing to provide network interconnection and Internet access.
By subnet calculation, we have the following subnet size
1st network: /27 subnet to cover 30 hosts
2nd network: /27 subnet to cover 20 hosts
3rd network: /28 subnet to cover 10 hosts
4th network: /29 subnet to cover 5 hosts
5th network: /29 subnet to cover 3 hosts
Server Farm: /28 subnet to cover 12 hosts
Between Router 4 and Firewall
Host # : 2
Smallest Subnet: /30
Between Routers
Host # : 4
Smallest Subnet: /29
Let's look at Router 1. Since there are multiple networks behind the Router 1, it is a good idea to have supernet on the router to represent all the networks behind it. Similar concept applies to Router 3 that have multiple networks as well.
To supernet, the smaller networks should be in consecutive order such as 192.168.0.0/27 and 192.168.0.32/27 for 1st and 2nd networks respectively.
Let's review supernet consideration at Router 1. There are following networks behind the router
1st network: 192.168.0.0/27 (192.168.0.1 - 192.168.0.30)
2nd network: 192.168.0.32/27 (192.168.0.33 - 192.168.0.62)
3rd network: 192.168.0.64/28 (192.168.0.65 - 192.168.0.78)
As mentioned, you can supernet both /27 networks to be a single /26 network which is 192.168.0.0/26. With this supernet choice, Router 1 represents the following networks to the rest of the organization.
192.168.0.0/26 (192.168.0.1 - 192.168.0.62)
192.168.0.64/28 (192.168.0.65 - 192.168.0.78)
Let's say you want Router 1 to represent only a single network to the rest of organization. You could choose to supernet all networks behind Router 1 into a single /25 network, which would be 192.168.0.0/25 (192.168.0.1 - 192.168.0.126).
When you do this /25 network supernet, then you have to make sure that the following subnets are only behind the Router 1.
192.168.0.80/28 (192.168.0.81 - 192.168.0.94)
192.168.0.96/27 (192.168.0.97 - 192.168.0.126)
If one of these networks are not behind the Router 1 and there is 192.168.0.0/25 supernet on Router 1, the one network could become unreachable from the rest of the organization; which could lead to unreliable network.
From scalability perspective, you need to forecast if the above assigned subnets are sufficient to support all the three networks. If the 3rd network host number will grow to 30 within 2 years let's say, then it is probably a good idea to assign /27 network instead of /28 network to the 3rd network to anticipate the growth. This way, the 3rd network subnet assignment would be good until at least 2 years in the future.
Static and Dynamic Routing
Let's review the following network design
Internet
|
|
Router 4
.13 |
192.168.0.12/30 |
.14 |
Router 3 --- 192.168.3.0/24
.1 / \ .6
192.168.0.0/30 / \ 192.168.0.4/30
.2 / \ .5
Router 1 Router 2
| |
192.168.1.0/24 192.168.2.0/24
The 192.168.1.0/24, 192.168.2.0/24, and 192.168.3.0/24 are broadcast networks. 192.168.0.0/28 is point-to-point network.
192.168.1.0/24 is behind Router 1. 192.168.2.0/24 is behind Router 2. 192.168.3.0/24 is behind Router 3.
192.168.0.1 and 192.168.0.6 are Router 3. 192.168.0.2 is Router 1. 192.168.0.5 is Router 2.
To access the Internet from 192.168.1.0/24, 192.168.2.0/24, and 192.168.3.0/24 networks; the outbound traffic must pass Router 4. Only Router 3 has direct connection to Router 4. To access the Internet from 192.168.1.0/24 and 192.168.2.0/24 networks, the outbound traffic must pass Router 3.
Note that static routing should be sufficient to provide reliable connection within the entire network. Followings are the static routes.
Router 1
Use 192.168.0.1 (Router 3) to reach any network outside 192.168.1.0/24
Router 2
Use 192.168.0.6 (Router 3) to reach any network outside 192.168.2.0/24
Router 3
Use 192.168.0.2 (Router 1) to reach 192.168.1.0/24
Use 192.168.0.5 (Router 2) to reach 192.168.2.0/24
Use 192.168.0.13 (Router 4) to reach any network outside 192.168.0.0/22
Router 4
Use 192.168.0.14 (Router 3) to reach 192.168.0.0/22
Use the ISP device IP address to reach the Internet (any network outside 192.168.0.0/22)
Let's say that Router 1 and Router 2 now have direct connection as follows
Internet
|
|
Router 4
.13 |
192.168.0.12/30 |
.14 |
Router 3 --- 192.168.3.0/24
.1 / \ .6
192.168.0.0/30 / \ 192.168.0.4/30
.2 / \ .5
Router 1 ------------ Router 2
| .9 .10 |
| 192.168.0.8/30 |
| |
192.168.1.0/24 192.168.2.0/24
where 192.168.0.9 is Router 1 and 192.168.0.10 is Router 2.
Let's look at the Router 1. From Router 1, it is possible to reach Router 3 directly or indirectly via Router 2. Similarly from Router 3, it is possible to reach Router 2 directly or indirectly via Router 1.
With this new connection, static routes no longer reliable choice. It is suggested that dynamic routing is used on Router 1 to 3. Router 4 may just keep using static route to the ISP device and to the Router 3.
When all of Router 1 - Router 3, Router 1 - Router 2, Router 2 - Router 3 connections have equivalent bandwidth and have equivalent traffic load; you may consider to use RIP. In real network, it is unlikely to have equivalent traffic load across the three connections. Therefore it is then suggested that Router 1 to Router 3 are to run OSPF, IS-IS, or EIGRP.
In Cisco routers, following is the EIGRP sample configuration
Router 1
router eigrp 10
network 192.168.1.0 255.255.255.0
network 192.168.0.0 255.255.255.252
network 192.168.0.8 255.255.255.252
Router 2
router eigrp 10
network 192.168.2.0 255.255.255.0
network 192.168.0.4 255.255.255.252
network 192.168.0.8 255.255.255.252
Router 3
router eigrp 10
redistribute static
network 192.168.3.0 255.255.255.0
network 192.168.0.0 255.255.255.248
The static routes
Router 3
ip route 0.0.0.0 0.0.0.0 192.168.0.13
Router 4
ip route 192.168.0.0 255.255.252.0 192.168.0.14
ip route 0.0.0.0 0.0.0.0 [ISP DEVICE IP ADDRESS]
WAN Design
Check out the following FAQ for insights
»Cisco Forum FAQ »Redundant Internet Link Graceful Load Balance/Failover
feedback form
by aryoba
last modified: 2007-07-07 07:02:16
FYI, your company may not run out IP address just yet. There might be something improper with the company's current network design. The following thread might give you a fresh insight.
»running out of IP addresses
Basically the sample company incorporate "flat network". I refer "flat network" as using the very same /24 subnet for all departments. As illustration, your company uses 10.10.10.0/24 subnet for all departments; where 10.10.10.2 can be in Sales, 10.10.10.5 can be in Accounting, 10.10.10.9 can be in IT, and so on.
As the thread shows, such usage of "flat network" is not suggested. When your company grows as the sample company, then you need to break up your company network into smaller logical network. As example, set 10.10.0.0/24 only for Sales; set 10.10.1.0/24 only for Accounting; and so on. Also as the thread shows, you need either a layer-3 switch or a router with a layer-2 switch that is capable to do inter-VLAN routing and trunking.
feedback formby aryoba
last modified: 2006-04-16 14:24:50
»Cisco Forum FAQ »Setting Up Private Site-To-Site Connections
Tips:
* Use private subnets whenever feasible
* If most of branches or departments within your organization already use let's say 192.168.x.x subnet, then keep using it. You may have renumber the 3rd octet to avoid overlap.
* Avoid discontiguous networks at all costs
* Assign separate subnets for infrastructure interconnections, servers, workstations, routers, switches, firewalls, IDS/IPS devices, UPS, and all other network devices
* When assigning subnets to network devices, plan network growth room for at least the next 5 years
* When there would be only two devices directly connected to each other, assign /30 or /31 subnet
* Avoid running dynamic routing protocols unless there are multiple exits with the equivalent administrative distances, connection technology, or bandwidth
* Should you decide to use OSPF, then keep in mind that you don't really need to have multiple areas unless you have a good reason to have so
* Consider resilient disaster recovery which might require layer-2 and layer-3 redundancies
* Do not "force" to have load balance over multiple links due to possible side effects of asymmetric routing
1. Hub and Spoke
Typical setup
* There is one site (probably the main or corporate office) that has direct connection to all other sites; called "Hub"
* All other sites (usually remote offices or branches) only has a single connection to the Hub; called "Spoke"
* Hub-to-Spoke communication use the direct connection
* Spoke-to-Spoke communication must go through the Hub as "intermediate hop"
* Connection to external network (i.e. the Internet) only exists at the Hub
* Communication between Spoke and external network must go through the Hub
Tips:
* Since from Spoke perspective, traffic must go through Hub to reach other sites or external network; a single static route as default gateway pointing to Hub should be sufficient to cover all communication type
* From Hub perspective, traffic must go through each dedicated connection to reach specific Spoke or external network; a single static route as default gateway pointing to the external network (i.e. the ISP) and several static routes to reach Spokes should be sufficient to cover all communication type
* No need to run dynamic routing
* To have more resilient connection, bonded circuits (i.e. bonded T1/E1 circuits) between Hub and Spokes can be considered. Other consideration is to have redundant circuits between Hub and Spokes that are served by multiple ISP
* Hub network device should be the most powerful one compared to the Spoke network device since Hub must support traffic from all Spokes and the external network where the Spoke only support traffic within itself
* Should there future need to have backup connection beyond bonded circuit, refer to the next setup
2. Full Mesh
Typical Setup
* There is probably no single main or corporate office as the Hub
* All sites have direct connection to all other sites
* All sites might have direct connection to external network (i.e. the Internet)
* Multiple path to reach the same site from one site perspective might exist
Tips:
* When there are multiple path to reach the same site, running dynamic routing protocol (i.e. OSPF or EIGRP) is highly recommended to have resilient or optimal connection
* All sites should have equivalent network device specification and circuit bandwidth to maintain predictable network behavior
3. Partially Mesh
Typical Setup
* There are probably at least two main sites (two "Hubs" that have direct connection to all other sites ("Spokes")
* All other sites (the Spokes) have one direct connection to 1st main site and another direct connection to 2nd main site
* Connection from Spoke to 1st main site is probably the preferred (primary) connection
* Connection from Spoke to 2nd main site is probably the alternate (backup) connection
* Connection between two main sites is probably using the most reliable and feasible connection, that could be in a form of single or multiple redundant connections
* Multiple path to reach other site from one site perspective exists
* Connection to external network (i.e. the Internet) might only exist at main site
* As illustration, the Spoke primary connection is in a form of Frame Relay circuit where the backup connection is in a form of ISDN (dialup) or broadband circuit; and connection between two main sites is in a form of bonded T1/E1 circuits
Tips:
* Since there are multiple path to reach the same site, running dynamic routing protocol (i.e. OSPF or EIGRP) is highly suggested to provide resilient and optimal connection
Network Topology Variations
* Hub and Spoke with one primary and one backup connections between Hub and all Spokes
* Each Spoke has direct connection to reach the external network (i.e. the Internet) without go through Hub
Running Dynamic Routing Protocol in Primary-Backup Connection Scenario
1. Both Primary and Backup Circuits are always up on flat rate from billing perspective
* At remote site, dynamic routing protocol run over both primary and backup circuit
* Primary circuit from the routing protocol perspective should have lower cost compared to the backup circuit
* When there are multiple main sites (the Hub and Spoke with multiple Hub), route from Spoke to Main Hub might be preferrable over route from Spoke to Secondary Hub
2. Only Primary Circuit is always up on flat rate from billing perspective
* To avoid the backup circuit goes up due to the Hello mechanism, no dynamic routing run over backup circuit; just static routes with higher administrative distance or metric than the dynamic routing protocol's
* When primary circuit terminates at different equipment than the backup circuit, there might be a need to redistribute the static route that run over backup circuit to the dynamic routing protocol domain with the purpose of introducing known alternate path
For illustrations, check out following threads:
»Pix 520 with multiple ISP connections?
»[Config] ISDN Multisite dialup using OSPF cost issue
feedback formby aryoba
last modified: 2007-07-19 11:40:52
Tips in connecting hosts to switches (#13583)
1. Group them in one VLAN as possible
Identify which computers that most of the time access particular servers. Then group all of those computers and the associated servers in one VLAN as possible.
2. Setup dedicated VLAN for servers only when necessary
When you have users from multiple departments accessing one specific server, then put this particular server in dedicated VLAN. Do not put regular non-server computer into this VLAN due to security vulnerability.
3. "Minimize Distance" between computers
Whenever possible; put all hosts (computers, servers, printers, etc.) in one single switch. When all of the hosts are spread into multiple switches, shorten the distance between switches. Especially for high traffic or critical application, use the largest bandwidth possible to "minimize distance" between hosts.
Illustration:
We use a switch with 24 ports of 10/100 Mbps speed and 2 ports of 10/100/1000 Mbps speed. Should we use the Gigabit port to connect to servers and use Fast Ethernet ports as trunk?
Answer:
When you have hosts spread into multiple switches, then you should use the Gigabit-capable port to connect to servers and 10 Gigabit-capable port as trunk (of course whenever possible).
Keep in mind that as general idea, should always use the fastest port available as trunk when there are hosts spread into multiple switches. Do not, as example, use Gigabit port for server connections and Fast Ethernet port as trunk in this multiple switches situation since the traffic across switches are bogged down at the trunk (the trunk causes "bottleneck"). It is then better to use the Fast Ethernet port for server and the Gigabit port for trunk.
feedback formby aryoba
last modified: 2005-12-31 20:33:27
Should I setup the network as following?
S1---S2---S3---S4---S5
|
Router
Answer:
Of course you can setup such design. However keep in mind that such design is a bad one. Why? Each link between one switch and another is single point of failure. If the link between S3 and S4 breaks; S5 cannot reach Router, S1, and S2. The same sense applies to link between S2 and S3, link between S1 and S2, and link between Router and S1.
Now to solve the failure link possibility, let me ask you this. How resilient do you wish your network be? How do you like your network design between the followings:
1)
Router
|
S1
|
+----+--+--+----+
| | | |
S2 S3 S4 S5
2)
Router
|
S1
|
+----+--+--+----+
| | | |
+-- S2 S3 --- S4 S5 --+
| | | | | |
| +----+ +----+ |
| |
+------------------------+
3)
Router
/ \
+--- S1 ---- S2 --+
| / | | \ |
| | +-- S3 -+ | +
| | | |
| +---- S4 ---+ |
| |
+------- S5 ------+
So far, the network resiliency issue discussed is just from the internal network (layer-2) point of view. There are also other issues to think of. The followings are just examples.
1. Redundant Routers
In case one router fails (either hardware or software failure).
2. Multiple ISP
When one ISP goes down or has bottleneck traffic issues, there is still another to load balance. You might want to have the 2nd ISP to use different telco lines (different local loop) that connects your site to theirs.
3. Having UPS (Uninterruptable Power Supply)
In case of power outage, there is still time to make necessary file access, save, or backups.
For further info of network resiliency, check out the following FAQ
»Cisco Forum FAQ »Network Design Tips
»Cisco Forum FAQ »Redundant Internet Link Graceful Load Balance/Failover
feedback formby aryoba
last modified: 2007-05-20 08:07:19
BGP Design (#14664)
»Cisco Forum FAQ »Redundant Internet Link Graceful Load Balance/Failover
Prepending Your AS to setup automatic BGP failover mechanism on remote router via iBGP
One of the BGP feature is AS prepending, done by utilizing the BGP AS_PATH attribute. Let's say your organization edge routers are running BGP with your ISP. There are multiple links connecting your edge routers to ISP routers. One link has larger bandwidth than another.
You would then prefer that incoming traffic from your ISP to your organization via your edge routers would consider link with larger bandwidth as primary path over another link. In other words, other link would be the secondary or backup link in case the primary link goes down.
By prepending your AS on the edge routers' secondary link and no AS prepend on the primary link, your ISP would see that it takes longer path to reach your organization via the secondary link over the primary link. Therefore your ISP would prefer to use the primary link for incoming traffic from the Internet to your organization.
Following is a sample configuration
| AS 100 | AS 300
| | I
| R1 ----+----- ISP-A ------ N
| | | T
| Host --------+ | E
| | | R
| R2 ----+----- ISP-B ------ N
| | E
| | AS 400 T
Assumption
• Your Public AS number is 100
• You have two Internet edge routers, R1 and R2
• R1 is BGP peering with ISP A, and R2 is BGP peering with ISP B
• ISP A Public AS number is 300 and ISP B Public AS number is 400
• Hosts, R1 Ethernet, and R2 Ethernet interfaces are within 192.168.21.0/24 subnet
• R1 and ISP-A Serial interfaces are within 192.168.31.0/30
• R2 and ISP-B Serial interfaces are within 192.168.42.0/30
• You prefer to have inbound traffic over ISP A instead of ISP B
• You also prefer to have outbound traffic over ISP A instead of ISP B
• To have such inbound traffic condition, you prepend your AS number once on R2
• To have such outbound traffic condition, you set R1 local preference higher than the R2 default local preference (which is 100)
In a real network, prepending your AS number on R2 more than once might be required to achieve the desired result. In addition, R1 and R2 might be running HSRP or similar redundancy technique. Check out the following link of HSRP-BGP Combination sample configuration.
»www.cisco.com/en/US/tech/tk365/t···2c.shtml
Reality Insight
In BGP theory, there are several options to setup automatic failover mechanism. Some of them are AS prepending, MED, and BGP Community utilization.
When you manage the entire BGP network (that have multiple AS numbers within a single network administration), then you can use any techniques to control the failover mechanism. This applies usually when you run internal BGP network within your organization.
Managing external BGP network with multiple providers (ISP) would be different story. Not all ISP honor MED. AS prepend on your own AS number (or the ISP AS number) may not affect the inbound traffic from the Internet where transit provider still see particular ISP as the most actractive connection.
Following threads are some discussions
»BGP Multihoming default-originate only setup
»BGP Question
»BGP
BGP Community utilization should be the best approach to control inbound and outbound traffic over multiple AS (including over multiple providers in external BGP network). When your ISPs provide decent BGP Community string, you will have a good automatic failover mechanism.
In addition, Cisco provide BGP Conditional Subnet Advertisement feature that might be useful as workaround when MED, AS prepend, and BGP Community do not fulfill your expectation. However this feature looks like still in a stage phase since there is still unresolved Cisco Bug ID relating to "uncooperative" subnet advertisement problem.
feedback formby aryoba
last modified: 2008-08-05 13:06:40
Migrating network (#14766)
»[Config] BGP and OSPF redistribution
feedback formby aryoba