This section contains instructions on how to install and configure the Top of Rack Switch (ToR Switch) inside the local FIBRE island, which is responsible for the control plane network of the local environment and to link the island to the FIBREnet backbone network.
ToR Switch Minimum Requirements
- A 24-port Gigabit Ethernet Switch with:
- Remote Management support - mandatory
- SNMPv2 enabled - mandatory
- 4094 VLANs - mandatory
- Q-in-Q support - mandatory
- Selective Q-in-Q support - desirable
- OpenFlow 1.0 support - desirable
A higher port density is desirable according to the quantity of resources the local FIBRE island will be offering.
A more basic switch, with VLANs support but no Q-in-Q required, can be used to extend the number of ports dedicated to island resources, through a cascade-style connection.
Models currently in operation:
- Datacom DM4100 48-port Switch
- Dell N2024 24-port Switch
- Models currently under homologation:
- Dell PowerConnect 5548 48-port Switch
- Extreme X440-G2-24p-10GE4 24-port Switch
General Topology
General topology and connections for an initial island setup:
General topology and connections for a complete island:
Interface Layout
Standard bayface and interfaces for the DM4100:
Procedure
Before the Switch configuration - Firmware upgrade:
To execute a local firmware upgrade, is necessary to configure a TFTP server on a notebook. It's recommended to use the “Tftpd32 4.0” software related in the example below. All the example is for the DM4100 Switch:
- Create a new folder, download and copy the new firmware file to it, from: ftp://200.130.15.187/Firmware_DM4100/
- Download and decompress the Tftpd32 server application: ftp://200.130.15.187/Tftpd32/
- Configure a secondary IP address in the notebook ethernet interface: 10.0.0.1/24.
- Execute the Tftpd32 server e configure it in the main screen:
Current Directory: [fill in wit the entire filepath for the new firmware file]
Server interface: 10.0.0.1 - Connect the notebook ethernet interface to the mgmt-eth interface on the Datacom switch (cat RJ-45).
Access the equipment console and execute the following commands (this example is for the firmware file “DM4100_48P_OF_1.0.8.im” – substitute the file name if applicable):
! configure ! interface vlan 1 ip address 10.0.0.2/24 no shutdown end ! copy tftp 10.0.0.1 DM4100_48P_OF_1.0.8.im firmware !
Verify if the new version is marked as the next firmware to load (S Flag):
! show firmware !
Execute an equipment reboot and verify wether the new version is active (R Flag):
! reboot ... show firmware !
- End
Configuration steps for the ToR Switch:
All the examples are for the DM4100 Switch.
The initial configuration assumes the island is connected via VPN.
The 11th step is a "migration" step, to switch the connection from the VPN to the dedicated backbone called FIBREnet:
- Hostname
- Timezone
- SSH server, Telnet and HTTP
- SNMP server
- Routing and Default Gateway
- NTP
- Interfaces
- VLANs
- STP
- Users
- VPN to FIBREnet migration
Hostname - replace teh "[island-name]" string by the institution acronym:
! hostname tor-[island-name].fibre.org.br !
Timezone - the clock timezone is set to UTF -3, with nem "BRT"
! clock timezone BRT -3 !
SSH server, Telnet and HTTP - is recommended to disable telnet access, as the SSH is the standard in FIBREs network.
! ip ssh host-key generate ! here a new key will be generated ! ! no ip telnet server ip http server ip http secure-server ! ip ssh server !
SNMP server - the SNMP server is configured with local operator's contact info, equipment location and the official FIBRE read-only comunity. It's recommended to disable the public community.
! ip snmp-server ip snmp-server contact [Operator's e-mail address, or fibre-rnp@rt.rnp.br] ip snmp-server location [Institution Acronnym, City, UF] no ip snmp-server community public ro ip snmp-server community FIBRE-BR ro !ip snmp-server community !@FIBRE-BR#$ ro !
Routing - initially, a default route to the VPN template is created in order to forward the internal traffic through the VPN gateway.
! ip routing ! ip route 10.0.0.0/8 10.[Institution-ID].0.70 !
NTP - the NTP server is configured for timestamp syncronization:
! sntp client sntp poll 3600 sntp server 10.128.0.150 !
VLANs - The [Institution-ID] number utilized in the IP addressing is defined in this document. The recommendation is to use the VLAN ID 60 for the island's control plane and VLAN 50 for the CMC icarus control network.
! vlan qinq ! interface vlan 10 name Internet set-member tagged ethernet 1/1 set-member untagged ethernet 1/2 ! interface vlan 50 name Rede CMC set-member untagged ethernet 1/14 set-member untagged ethernet 1/16 set-member untagged ethernet 1/18 set-member untagged ethernet 1/20 set-member untagged ethernet 1/22 ! interface vlan 60 name Rede Controle ip address 10.[Institution-ID].0.1/24 set-member tagged ethernet 1/1 set-member untagged ethernet 1/4 set-member untagged ethernet 1/5 set-member untagged ethernet 1/6 set-member untagged ethernet 1/7 set-member untagged ethernet 1/11 set-member untagged ethernet 1/12 set-member untagged ethernet 1/13 set-member untagged ethernet 1/15 set-member untagged ethernet 1/17 set-member untagged ethernet 1/19 set-member untagged ethernet 1/21 ! interface vlan 3220 name FIBREnet Controle no ip address set-member tagged ethernet 1/47 ! interface vlan 3221 name FIBREnet Dados set-member untagged ethernet 1/3 set-member tagged ethernet 1/48 ! vlan-group 1 vlan-group 1 vlan all !
Interfaces:
! interface ethernet 1/1 description VMServer1 controle no shutdown ! interface ethernet 1/2 description Internet Uplink switchport native vlan 10 no shutdown ! interface ethernet 1/3 description OFSwitch1 dados (pronto) switchport native vlan 3221 no shutdown ! interface ethernet 1/4 description OFSwitch1 c (pronto) switchport native vlan 60 no shutdown ! interface ethernet 1/5 description OFSwitch2 c (netfpga1) switchport native vlan 60 no shutdown ! interface ethernet 1/6 description OFSwitch3 c (netfpga2) switchport native vlan 60 no shutdown ! interface ethernet 1/7 description OFSwitch4 c (netfpga3) switchport native vlan 60 no shutdown ! interface ethernet 1/8 shutdown ! interface ethernet 1/9 shutdown ! interface ethernet 1/10 shutdown ! interface ethernet 1/11 description Perfsonar int1 switchport native vlan 60 no shutdown ! interface ethernet 1/12 description Perfsonar int2 switchport native vlan 60 no shutdown ! interface ethernet 1/13 description icarus1 c switchport native vlan 60 no shutdown ! interface ethernet 1/14 description icarus1 cmc switchport native vlan 50 no shutdown ! interface ethernet 1/15 description icarus2 c switchport native vlan 60 no shutdown ! interface ethernet 1/16 description icarus2 c switchport native vlan 50 no shutdown ! interface ethernet 1/17 description icarus3 cmc switchport native vlan 60 no shutdown ! interface ethernet 1/18 description icarus3 c switchport native vlan 50 no shutdown ! interface ethernet 1/19 description icarus4 c switchport native vlan 60 no shutdown ! interface ethernet 1/20 description icarus4 cmc switchport native vlan 50 no shutdown ! interface ethernet 1/21 description icarus5 c switchport native vlan 60 no shutdown ! interface ethernet 1/22 description icarus5 cmc switchport native vlan 50 no shutdown ! interface ethernet 1/23 shutdown ! interface ethernet 1/24 shutdown ! interface ethernet 1/25 shutdown ! interface ethernet 1/26 shutdown ! interface ethernet 1/27 shutdown ! interface ethernet 1/28 shutdown ! interface ethernet 1/29 shutdown ! interface ethernet 1/30 shutdown ! interface ethernet 1/31 shutdown ! interface ethernet 1/32 shutdown ! interface ethernet 1/33 shutdown ! interface ethernet 1/34 shutdown ! interface ethernet 1/35 shutdown ! interface ethernet 1/36 shutdown ! interface ethernet 1/37 shutdown ! interface ethernet 1/38 shutdown ! interface ethernet 1/39 shutdown ! interface ethernet 1/40 shutdown ! interface ethernet 1/41 shutdown ! interface ethernet 1/42 shutdown ! interface ethernet 1/43 shutdown ! interface ethernet 1/44 shutdown ! interface ethernet 1/45 shutdown ! interface ethernet 1/46 shutdown ! interface ethernet 1/47 description FIBREnet Controle no shutdown ! interface ethernet 1/48 description FIBREnet Dados no shutdown !
STP (Spanning Tree configuration for the FIBREnet standard is not used anymore):
! !spanning-tree 1 !spanning-tree 1 vlan-group 1 !
Users - is mandatory to create the default FIBRE users within the equiment and their respective perimissions. This information can be requested by e-mail along the island installation. Furthermore, the local island team is allowed to create any other user account, according to the local policies.
! ! request user info via e-mail and create users !
VPN to FIBREnet migration - skip this step if the island is going to use a VPN connection. These configurations needs to be applied in order to migrate the island federal connection from the VPN to the dedicated backbone links known as FIBREnet. A Default Gateway can be configured in this step, if the local network uses a firewall or another gateway for internet connection. This changes in the ToR Switch has to be made in conjunction with another configuration changes in Dom0 following the same objective. It's recommended to access the equipments via console or the island network (10.[Institution-ID].0.1) to prevent any loss of session connectivity during the activity.
ip default-gateway [internet-gateway-ip-address] ! ip route 10.0.0.0/8 192.168.0.136 no ip route 10.0.0.0/8 10.[institution-ID].0.70 ! interface vlan 3220 ip address 192.168.0.[Institution-ID]/24 !
Validation, Backup and Saving
It's recommended to save the configuration already made at this point, in a separate file for backup and in the switch's startup configuration file:
! show firmware ! show running-config ! ! Salvar a saída num arquivo texto e guardá-lo como backup ! Salvar a configuração na startup-config com o comando: ! copy running-config startup-config !