pimpmynetwork.org |
Global Internet Exchange
 |
|
|
|||
|
Version 0.2.0 Contents 1. Introduction The this document is intended to be an exercise in the creation of a commercial global peering platform using an MPLS network to create a global VPLS instance across which traffic can be passed between peers on metro ethernet networks at layer 2.
Diagram showing how a network capable of supporting the peering platform would look showing the VPLS cloud, MPLS LSRs and individual metro ethernet rings The Basics There exist today numerous Internet Exchanges around the world. They're predominantly ethernet-based, such as the LINX, DE-CIX and AMSIX. Ethernet has been standardised upon because of its low cost, reliability, ease of implementation and almost unmatched speeds. Scalability is of paramount importance to most existing exchanges, as their traffic has been growing enourmously over the past 12 months and continues to do so. Because of the ubiquity of Ethernet, we'll use it as the access media. The MPLS network may be constructed of either Ethernet or SDH technologies, depending upon the distances involved and the amount of traffic the platform is required to carry. Most IXes have traditionally left the session configuration down to individual members, with members contacting eachother and configuring individual sessions between eachother. Many IXes have also implemented Multi-Lateral Peering (MLP), where each member peers with a route-server which automatically distributes their routes to each other member which has peered with the route-server. As this peering platform is commercial and we want as much traffic as possible to flow across it, it therefore makes sense to take the MLP approach, and require the use of route-servers. The Design The obvious design for the platform is to take a globally provisioned ethernet VPLS instance as the base and configure two or more route servers to distribute routes across the network. It's a relatively standard design, very much following the form of existing metro-scale peering platforms. Additional security could be added to the platform by introducing prefix-lists either statically maintained, or automatically generated from RIPE or another RIR's routing registry. Customers could also be restricted to only establishing peering sessions with the route-server. This basic design allows each customer to send/recieve routes to/from each other customer. The Flaws In The Design Several serious flaws will become apparent with the design once a number of peers are connected to it. A primary flaw is the lack of flexibility in route-selection due to limitations in BGP when using traditional implementations. Lets use AS286 as an example. AS286 is present in both London and Frankfurt and they announce the same prefixes to the route-server from both locations. When the route-servers recieve the prefixes from the London and Frankfurt peers, they use the standard BGP route-selection process to chose a single best route for each prefix. This means that one of the locations will never be used as a destination for traffic, because it will always lose the route-selection process! The issue can be further higlighted using AS2000 and AS286 an example. AS286 sends us a route in Frankfurt and London, the Frankfurt route takes precedence and is sent to all peers. AS20000 in both London and Amsterdam then starts sending traffic across europe to reach AS286 in Frankfurt, ignoring London! Similarly, routes announced from AS2000 in Amsterdam may win the route-selection process, and recieve traffic from both the London and Frankfurt AS286 peers. As the diagram below shows, this would cause unnecessary utilisation of costly intercity and international connectivity rather than relatively cheap metro connectivity.
Diagram showing non-optimal traffic flow between AS20000 and AS286 The problem would cause similar problems for peers present on different continents. Assume AS286 is present in Seattle and London and AS20000 is present in Frankfurt. If the Seattle route took precedence over the London route, AS20000 would send its traffic all the way to Seattle to reach AS286, rather than taking the much shorter London route! A number of things have been assumed :
In order to achieve this, we need to do a number of things on the route-servers:
In order to provide peers with the information they need to filter routes and to provide the route-server with the information it needs to build the individual RIBs, we'll tag each route as it's announced to us with a series of communities : 45666:<Peer AS> For example, a peer in London would be tagged with the following communities : 45666:8575 The downstream peer could then use this information to raise or lower the preference of routes based on their geographic origin. For example, a peer may not want to use any routes coming from the same country as themselves, so would filter routes with the community 65000:65000 if they were in the UK.
Example of route-maps tagging ingress routes The use of multiple routing information bases (RIBs) within the route server is the most important correction to be made. As previously explained, a traditional BGP speaker has only a single 'live' BGP database, and it can only store a single version of each prefix in that database. This causes the entire peering platform's BGP configuration to be sub-optimal. This was clearly demonstrated in the above examples and diagram. This is solved by creating a seperate routing table for each peer connected to the route-server, so that routes can be filtered and preferenced for each peer before the routes are announced. This must be done on the route-server because BGP is unable to transmit two versions of the same prefix, any new version merely over-writes the previous one. Multiple RIBs allow us to build complex filters within the route-server to select routes based on their suitability for the peer they'll be announced to using information from communities imposed at ingress. This means that provided we have suitably configured filters, peers will always talk to the closest exit point to them, approximating hot-potato routing.
Flowchart showing best-path selection based on community filters.
5. Egress Engineering Filtering For much the same reasons that we'd provide peers with the information to filter the routes we advertise to them, we'd provide them with a mechanism to prevent their routes being used by peers they don't want to reach across the peering platform. This could take the form of completely blocking the announcement based on peer as, region, country or city, or of prepending announcements based on the same attributes. For example, to block announcing a route to an entire geographic region the peer would tag their routes with a community of the form 64999:<region id>. They could then selectively allow announcements to cities or countries by using a community of the form 64888:<city/contry id>.
Flowchart showing filtering process before routes are weighed for announcement This allows for a great deal of flexibility for the customer without the intervention of the platform's engineering resource.
An example of egress filters for a london based peer, AS8575. The system could be extended further to allow prepending of the peering point's AS onto announcements perhaps using 65001:<id> to prepend once and 65003:<id> to prepend three times. The configuration below shows neighbour statements, community-lists, prefix-lists and route-maps for an imaginary peer with as 8575 located in London, UK.
A complete configuration example for peer AS8575 The diagram below charts the flow of prefixes through the various route-maps required to create the desired peer-specific routing tables.
Diagram showing the complete path of prefixes as they're evaluated for AS8575's RIB We've now provided our customers both with an optimal routing table based on their geographic location and the tools to fully customise their service and enforce their own peering policies. This could help overcome some resistance from organisations with non-open peering policies This allows for people to connect to the peering platform but not automatically accept routes from or announce their routes to any other peer. This would be achieved by using the global deny community on their outgoing announcements, and selectively adding communities identifying locations or individual peers they would like to peer with, and configuring ingress filters on their own routers denying all prefixes not bearing communities identifying the areas or peers they wish to peer with.
Diagram showing optimal traffic flow across the peering platform after sufficient filtering has been applied Additional Considerations As this platform is a commercial operation, it'll be charged accordingly. Transporting traffic from Frankfurt to Seattle is a costly business, and sufficient profit will have to be made to justify the creation of the platform. However, many potential customers of the peering platform might baulk at the prospect of paying international prices for traffic merely traversing the UK or Frankfurt metro rings, or even for traffic going from Amsterdam to Frankfurt. To remove this problem, the peering platform could be split up into two or three components:
I would envisage dedicated either a single dedicated port for low usage or multiple ports for high usage customers being used for all three services. This would allow a tiered based billing system to be used, where base the entire port usage has a low cost per mbps, the regional port usage attracts a premium in addition to the base usage totalling less than transit costs, and a premium for international traffic totalling equal to transit. I believe this would create a product capable of appealing to the widest range of potential customers possible.
Diagram showing the completed, multi-tiered peering platform in operation. Not all traffic has been shown for simplicity.
|
|
|||
|
|
|
