EtherChannel and IEEE 802.3ad Link Aggregation

EtherChannel and IEEE 802.3ad Link Aggregation

EtherChannel and IEEE 802.3ad Link Aggregation are network port aggregation technologies that allow several Ethernet adapters to be aggregated together to form a single pseudo Ethernet device. For example, ent0 and ent1 can be aggregated into an EtherChannel adapter called ent3; interface en3 would then be configured with an IP address. The system considers these aggregated adapters as one adapter. Therefore, IP is configured over them as over any Ethernet adapter. In addition, all adapters in the EtherChannel or Link Aggregation are given the same hardware (MAC) address, so they are treated by remote systems as if they were one adapter. Both EtherChannel and IEEE 802.3ad Link Aggregation require support in the switch so it is aware which switch ports should be treated as one.

The main benefit of EtherChannel and IEEE 802.3ad Link Aggregation is that they have the network bandwidth of all of their adapters in a single network presence. If an adapter fails, network traffic is automatically sent on the next available adapter without disruption to existing user connections. The adapter is automatically returned to service on the EtherChannel or Link Aggregation when it recovers.
There are some differences between EtherChannel and IEEE 802.3ad Link Aggregation. Consider the differences given in Table 3 to determine which would be best for your situation.

Table 3. Differences between EtherChannel and IEEE 802.3ad Link Aggregation.

EtherChannel	IEEE 802.3ad
Requires switch configuration	Little, if any, configuration of switch required to form aggregation. Some initial setup of the switch may be required.
Supports different packet distribution modes	Supports only standard distribution mode

Beginning with AIX 5L with 5200-03, Dynamic Adapter Membership functionality is available. This functionality allows you to add or remove adapters from an EtherChannel without having to disrupt any user connections. For more details, see Dynamic Adapter Membership.

Supported Adapters

EtherChannel and IEEE 802.3ad Link Aggregation are supported on the following Ethernet adapters:

• 10/100 Mbps Ethernet PCI Adapter
• Universal 4-Port 10/100 Ethernet Adapter
• 10/100 Mbps Ethernet PCI Adapter II
• 10/100/1000 Base-T Ethernet PCI Adapter
• Gigabit Ethernet-SX PCI Adapter
• 10/100/1000 Base-TX Ethernet PCI-X Adapter
• Gigabit Ethernet-SX PCI-X Adapter
• 2-port 10/100/1000 Base-TX Ethernet PCI-X Adapter
• 2-port Gigabit Ethernet-SX PCI-X Adapter

Only the basic EtherChannel functionality (operating exclusively in "standard" or "round-robin" mode without a backup) is supported in the following Ethernet adapters:

• PCI Ethernet BNC/RJ-45 Adapter
• PCI Ethernet AUI/RJ-45 Adapter

Unless the AIX Release Notes specify otherwise, support for new adapters will be provided as those adapters are released.

Note:

Mixing adapters of different speeds in the same EtherChannel, even if one of them is operating as the backup adapter, is not officially supported. This does not mean that such configurations will not work. The EtherChannel driver will make every reasonable attempt to work even in a mixed-speed scenario.

For information on configuring and using EtherChannel, see EtherChannel. For more information on configuring and using IEEE 802.3ad Link Aggregation, see IEEE 802.3ad Link Aggregation. For information on the different AIX and switch configuration combinations and the results they will produce, see Interoperability Scenarios.

EtherChannel

The adapters that belong to an EtherChannel must be connected to the same EtherChannel-enabled switch. This switch must be manually configured to treat the ports that belong to the EtherChannel as an aggregated link. Note that your switch documentation may refer to this capability as "link aggregation" or "trunking."
Traffic is distributed across the adapters in either the standard way (where the adapter over which the packets are sent is chosen depending on an algorithm) or on a round-robin basis (where packets are sent evenly across all adapters). Incoming traffic is distributed in accordance to the switch configuration and is not controlled by the EtherChannel operation mode.
In AIX, you can configure multiple EtherChannels per system, but it is required that all the links in one EtherChannel are attached to a single switch. Because the EtherChannel cannot be spread across two switches, the entire EtherChannel is lost if the switch is unplugged or fails. To solve this problem, a new backup option available in AIX 5.2 and later keeps the service running when the main EtherChannel fails. The backup and EtherChannel adapters should be attached to different network switches, which must be inter-connected for this setup to work properly. In the event that all of the adapters in the EtherChannel fail, the backup adapter will be used to send and receive all traffic. When any link in the EtherChannel is restored, the service is moved back to the EtherChannel.
For example, ent0 and ent1 could be configured as the main EtherChannel adapters, and ent2 as the backup adapter, creating an EtherChannel called ent3. Ideally, ent0 and ent1 would be connected to the same EtherChannel-enabled switch, and ent2 would be connected to a different switch. In this example, all traffic sent over en3 (the EtherChannel's interface) would be sent over ent0 or ent1 by default (depending on the EtherChannel's packet distribution scheme), whereas ent2 will be idle. If at any time both ent0 and ent1 fail, all traffic would be sent over the backup adapter, ent2. When either ent0 or ent1 recover, they will once again be used for all traffic.
Network Interface Backup, a mode of operation available for EtherChannel in AIX 4.3.3 and AIX 5.1, protects against a single point of Ethernet network failure. No special hardware is required to use Network Interface Backup, but the backup adapter should be connected a separate switch for maximum reliability. In Network Interface Backup mode, only one adapter at a time is actively used for network traffic. The EtherChannel tests the currently-active adapter and, optionally, the network path to a user-specified node. When a failure is detected, the next adapter will be used for all traffic. Network Interface Backup provides detection and failover with no disruption to user connections. Network Interface Backup was originally implemented as a mode in the EtherChannel SMIT menu. In AIX 5.2 and later, the backup adapter provides the equivalent function, so the mode was eliminated from the SMIT menu. To configure network interface backup in AIX 5.2 and later, see Configure Network Interface Backup.

Configuring EtherChannel

Follow these steps to configure an EtherChannel.

Considerations

• You can have up to eight primary Ethernet adapters and only one backup Ethernet adapter per EtherChannel.
• You can configure multiple EtherChannels on a single system, but each EtherChannel constitutes an additional Ethernet interface. The no command option, ifsize, may need to be increased to include not only the Ethernet interfaces for each adapter, but also any EtherChannels that are configured. In AIX 5.2 and earlier, the default ifsize is eight. In AIX 5.2 and later, the default size is 256.
• You can use any supported Ethernet adapter in an EtherChannel (see Supported Adapters). However, the Ethernet adapters must be connected to a switch that supports EtherChannel. See the documentation that came with your switch to determine if it supports EtherChannel (your switch documentation may refer to this capability also as link aggregation or trunking).
• All adapters in the EtherChannel should be configured for the same speed (100 Mbps, for example) and should be full duplex.
• The adapters used in the EtherChannel cannot be accessed by the system after the EtherChannel is configured. To modify any of their attributes, such as media speed, transmit or receive queue sizes, and so forth, you must do so before including them in the EtherChannel.
• The adapters that you plan to use for your EtherChannel must not have an IP address configured on them before you start this procedure. When configuring an EtherChannel with adapters that were previously configured with an IP address, make sure that their interfaces are in the detach state. The adapters to be added to the EtherChannel cannot have interfaces configured in the up state in the Object Data Manager (ODM), which will happen if their IP addresses were configured using SMIT. This may cause problems bringing up the EtherChannel when the machine is rebooted because the underlying interface is configured before the EtherChannel with the information found in ODM. Therefore, when the EtherChannel is configured, it finds that one of its adapters is already being used. To change this, before creating the EtherChannel, type smit chinet, select each of the interfaces of the adapters to be included in the EtherChannel, and change its state value to detach. This will ensure that when the machine is rebooted the EtherChannel can be configured without errors. For more information about ODM, see Object Data Manager (ODM) in AIX 5L Version 5.2 General Programming Concepts: Writing and Debugging Programs.
• If you will be using 10/100 Ethernet adapters in the EtherChannel, you may need to enable link polling on those adapters before you add them to the EtherChannel. Type smit chgenet at the command line. Change the Enable Link Polling value to yes, and press Enter.
  Note:
  In AIX 5L with 5200-03 and later, enabling the link polling mechanism is not necessary. The link poller will be started automatically.
• If you plan to use jumbo frames, you may need to enable this feature in every adapter before creating the EtherChannel and in the EtherChannel itself. Type smitty chgenet at the command line. Change the Enable Jumbo Frames value to yes and press Enter. Do this for every adapter for which you want to enable Jumbo Frames. You will enable jumbo frames in the EtherChannel itself later.
  Note:
  In AIX 5.2 and later, enabling the jumbo frames in every underlying adapter is not necessary once it is enabled in the EtherChannel itself. The feature will be enabled automatically if you set the Enable Jumbo Frames attribute to yes.

Configure an EtherChannel

1. Type smit etherchannel at the command line.
2. Select Add an EtherChannel / Link Aggregation from the list and press Enter.
3. Select the primary Ethernet adapters that you want on your EtherChannel and press Enter. If you are planning to use EtherChannel backup, do not select the adapter that you plan to use for the backup at this point. The EtherChannel backup option is available in AIX 5.2 and later.
   Note:
   The Available Network Adapters displays all Ethernet adapters. If you select an Ethernet adapter that is already being used (has an interface defined), you will get an error message. You first need to detach this interface if you want to use it.
4. Enter the information in the fields according to the following guidelines:
   • EtherChannel / Link Aggregation Adapters: You should see all primary adapters that you are using in your EtherChannel. You selected these adapters in the previous step.
   • Enable Alternate Address: This field is optional. Setting this to yes will enable you to specify a MAC address that you want the EtherChannel to use. If you set this option to no, the EtherChannel will use the MAC address of the first adapter.
   • Alternate Address: If you set Enable Alternate Address to yes, specify the MAC address that you want to use here. The address you specify must start with 0x and be a 12-digit hexadecimal address (for example, 0x001122334455).
   • Enable Gigabit Ethernet Jumbo Frames: This field is optional. In order to use this, your switch must support jumbo frames. This will only work with a Standard Ethernet (en) interface, not an IEEE 802.3 (et) interface. Set this to yes if you want to enable it.
   • Mode: You can choose from the following modes:
     • standard: In this mode the EtherChannel uses an algorithm to choose which adapter it will send the packets out on. The algorithm consists of taking a data value, dividing it by the number of adapters in the EtherChannel, and using the remainder (using the modulus operator) to identify the outgoing link. The Hash Mode value determines which data value is fed into this algorithm (see the Hash Mode attribute for an explanation of the different hash modes). For example, if the Hash Mode is standard, it will use the packet's destination IP address. If this is 10.10.10.11 and there are 2 adapters in the EtherChannel, (1 / 2) = 0 with remainder 1, so the second adapter is used (the adapters are numbered starting from 0). The adapters are numbered in the order they are listed in the SMIT menu. This is the default operation mode.
     • round_robin: In this mode the EtherChannel will rotate through the adapters, giving each adapter one packet before repeating. The packets may be sent out in a slightly different order than they were given to the EtherChannel, but it will make the best use of its bandwidth. It is an invalid combination to select this mode with a Hash Mode other than default. If you choose the round-robin mode, leave the Hash Mode value as default.
     • netif_backup: This option is available only in AIX 5.1 and AIX 4.3.3. In this mode, the EtherChannel will activate only one adapter at a time. The intention is that the adapters are plugged into different Ethernet switches, each of which is capable of getting to any other machine on the subnet or network. When a problem is detected either with the direct connection (or optionally through the inability to ping a machine), the EtherChannel will deactivate the current adapter and activate a backup adapter. This mode is the only one that makes use of the Internet Address to Ping, Number of Retries, and Retry Timeout fields. Network Interface Backup Mode does not exist as an explicit mode in AIX 5.2 and later. To enable Network Interface Backup Mode in AIX 5.2 and later, you must configure one adapter in the main EtherChannel and a backup adapter. For more information, see Configure Network Interface Backup.
     • 8023ad: This options enables the use of the IEEE 802.3ad Link Aggregation Control Protocol (LACP) for automatic link aggregation. For more details about this feature, see IEEE 802.3ad Link Aggregation.
   • Hash Mode: You can choose from the following hash modes, which will determine which data value will be used by the algorithm to determine the outgoing adapter:
     • default: In this hash mode the destination IP address of the packet will be used to determine the outgoing adapter. For non-IP traffic (such as ARP), the last byte of the destination MAC address is used to do the calculation. This mode will guarantee packets are sent out over the EtherChannel in the order they were received, but it may not make full use of the bandwidth.
     • src_port: In this hash mode the source UDP or TCP port value of the packet will be used to determine the outgoing adapter. If the packet is not UDP or TCP traffic, the last byte of the destination IP address will be used. If the packet is not IP traffic, the last byte of the destination MAC address will be used.
     • dst_port: In this hash mode the destination UDP or TCP port value of the packet will be used to determine the outgoing adapter. If the packet is not UDP or TCP traffic, the last byte of the destination IP will be used. If the packet is not IP traffic, the last byte of the destination MAC address will be used.
     • src_dst_port: In this hash mode both the source and destination UDP or TCP port values of the packet will be used to determine the outgoing adapter (specifically, the source and destination ports are added and then divided by two before being fed into the algorithm). If the packet is not UDP or TCP traffic, the last byte of the destination IP will be used. If the packet is not IP traffic, the last byte of the destination MAC address will be used. This mode can give good packet distribution in most situations, both for clients and servers.
       Note:
       It is an invalid combination to select a Hash Mode other than default with a Mode of round_robin.
     To learn more about packet distribution and load balancing, see Load-balancing options.
   • Backup Adapter: This field is optional. Enter the adapter that you want to use as your EtherChannel backup. EtherChannel backup is available in AIX 5.2 and later.
   • Internet Address to Ping: This field is optional and only takes effect if you are running Network Interface Backup mode or if you have only one adapter in the EtherChannel and a backup adapter. The EtherChannel will ping the IP address or host name that you specify here. If the EtherChannel is unable to ping this address for the Number of Retries times in Retry Timeout intervals, the EtherChannel will switch adapters.
   • Number of Retries: Enter the number of ping response failures that are allowed before the EtherChannel switches adapters. The default is three. This field is optional and valid only if you have set an Internet Address to Ping.
   • Retry Timeout: Enter the number of seconds between the times when the EtherChannel will ping the Internet Address to Ping. The default is one second. This field is optional and valid only if you have set an Internet Address to Ping.
5. Press Enter after changing the desired fields to create the EtherChannel.
6. Configure IP over the newly-created EtherChannel device by typing smit chinet at the command line.
7. Select your new EtherChannel interface from the list.
8. Fill in all the required fields and press Enter.

Configure Network Interface Backup

Network Interface Backup protects against a single point of network failure by providing failure detection and failover with no disruption to user connections. When operating in this mode, only one adapter is active at any given time. If the active adapter fails, another adapter in the EtherChannel will be used for all traffic. When operating in Network Interface Backup mode, it is not necessary to connect to EtherChannel-enabled switches.
The Network Interface Backup setup is most effective when the adapters are connected to different network switches, as this provides greater redundancy than connecting all adapters to one switch. When connecting to different switches, make sure there is a connection between the switches. This provides failover capabilities from one adapter to another by ensuring that there is always a route to the currently-active adapter.
In releases prior to AIX 5.2, Network Interface Backup mode was implemented as an explicit mode of operation in the EtherChannel SMIT menu. In AIX 5.2 and later, however, the backup adapter functionality provides the equivalent behavior, so the mode was eliminated from the SMIT menu.
Additionally, AIX 5.2 and later versions provide priority, meaning that the adapter configured in the primary EtherChannel will be used preferentially over the backup adapter. As long as the primary adapter is functional, it will be used. This contrasts from the behavior of Network Interface Backup mode in releases prior to AIX 5.2, where the backup adapter was used until it also failed, regardless of whether the primary adapter had already recovered.
For example, ent0 could be configured as the main adapter, and ent2 as the backup adapter, creating an EtherChannel called ent3. Ideally, ent0 and ent2 would be connected to two different switches. In this example, all traffic sent over en3 (the EtherChannel's interface) would be sent over ent0 by default, whereas ent2 will be idle. If at any time ent0 fails, all traffic would be sent over the backup adapter, ent2. When ent0 recovers, it will once again be used for all traffic.
While operating in Network Interface Backup Mode, it is also possible to configure the EtherChannel to detect link failure and network unreachability. To do this, specify the IP address or host name of a remote host where connectivity should always be present. The EtherChannel will periodically ping this host to determine whether there is still a network path to it. If a specified number of ping attempts go unanswered, the EtherChannel will fail over to the other adapter in the hope that there is a network path to the remote host through the other adapter. In this setup, not only should every adapter be connected to a different switch, but each switch should also have a different route to the host that is pinged.
This ping feature is only available in Network Interface Backup mode. However, in AIX 5.2 and later, if there is a failover due to unanswered pings on the primary adapter, the backup adapter will remain the active channel as long as it is working. There is no way of knowing, while operating on the backup adapter, whether it is possible to reach the host being pinged from the primary adapter. To avoid failing over back and forth between the primary and the backup, it will simply keep operating on the backup (unless the pings go unanswered on the backup adapter as well, or if the backup adapter itself fails, in which case it would fail over to the primary adapter). However, if the failover occurred because the primary adapter failed (not because the pings went unanswered), the EtherChannel will then come back to the primary adapter as soon it has come back up, as usual.
To configure Network Interface Backup in AIX 5.2, see Configure Network Interface Backup in AIX 5.2 and later. To configure Network Interface Backup in previous versions of AIX, see Appendix B. Configure Network Interface Backup in previous AIX versions

Configure Network Interface Backup in AIX 5.2 and later

1. With root authority, type smit etherchannel on the command line.
2. Select Add an EtherChannel / Link Aggregation from the list and press Enter.
3. Select the primary Ethernet adapter and press Enter. This is the adapter that will be used until it fails.
   Note:
   The Available Network Adapters displays all Ethernet adapters. If you select an Ethernet adapter that is already being used, you will get an error message and will need to detach this interface before you can use it. See the ifconfig command for information on how to detach an interface.
4. Enter the information in the fields according to the following guidelines:
   • EtherChannel / Link Aggregation Adapters: You should see the primary adapter you selected in the previous step.
   • Enable Alternate Address: This field is optional. Setting this to yes will enable you to specify a MAC address that you want the EtherChannel to use. If you set this option to no, the EtherChannel will use the MAC address of the primary adapter.
   • Alternate Address: If you set Enable Alternate Address to yes, specify the MAC address that you want to use here. The address you specify must start with 0x and be a 12-digit hexadecimal address (for example 0x001122334455).
   • Enable Gigabit Ethernet Jumbo Frames: This field is optional. In order to use this, your switch must support jumbo frames. This will only work with a Standard Ethernet (en) interface, not an IEEE 802.3 (et) interface. Set this to yes if you want to use it.
   • Mode: It is irrelevant which mode of operation you select because there is only one adapter in the main EtherChannel. All packets will be sent over that adapter until it fails. There is no netif_backup mode because that mode can be emulated using a backup adapter.
   • Hash Mode: It is irrelevant which hash mode you select because there is only one adapter in the main EtherChannel. All packets will be sent over that adapter until it fails.
   • Backup Adapter: Enter the adapter that you want to be your backup adapter. After a failover, this adapter will be used until the primary adapter recovers. It is recommended to use the preferred adapter as the primary adapter.
   • Internet Address to Ping: The field is optional. The EtherChannel will ping the IP address or host name that you specify here. If the EtherChannel is unable to ping this address for Number of Retries times in Retry Timeout intervals, the EtherChannel will switch adapters.
   • Number of Retries: Enter the number of ping response failures that are allowed before the EtherChannel switches adapters. The default is three. This field is optional and valid only if you have set an Internet Address to Ping.
   • Retry Timeout: Enter the number of seconds between the times when the EtherChannel will ping the Internet Address to Ping. The default is one second. This field is optional and valid only if you have set an Internet Address to Ping.
5. Press Enter after changing the desired fields to create the EtherChannel.
6. Configure IP over the newly-created interface by typing smit chinet at the command line.
7. Select your new EtherChannel interface from the list.
8. Fill in all the required fields and press Enter.

For additional tasks that can be performed after the EtherChannel is configured, see Managing EtherChannel and IEEE 802.3ad Link Aggregation.

Load-balancing options

There are two load balancing methods for outgoing traffic in EtherChannel, as follows: round-robin, which spreads the outgoing traffic evenly across all the adapters in the EtherChannel; and standard, which selects the adapter using an algorithm. The Hash Mode parameter determines which numerical value is fed to the algorithm.
The following table summarizes the valid load balancing option combinations offered.

Table 4. Mode and Hash Mode combinations and the outgoing traffic distributions each will produce.

Mode	Hash Mode	Outgoing Traffic Distribution
standard or 8023ad	default	The traditional AIX behavior. The adapter selection algorithm uses the last byte of the destination IP address (for TCP/IP traffic) or MAC address (for ARP and other non-IP traffic). This mode is typically a good initial choice for a server with a large number of clients.
standard or 8023ad	src_dst_port	The outgoing adapter path is selected by an algorithm using the combined source and destination TCP or UDP port values. Since each connection has a unique TCP or UDP port, the three port-based hash modes provide additional adapter distribution flexibility when there are several, separate TCP or UDP connections between an IP address pair.
standard or 8023ad	src_port	The adapter selection algorithm uses the source TCP or UDP port value. In the netstat -an command output, the port is the TCP/IP address suffix value in the Local column.
standard or 8023ad	dst_port	The outgoing adapter path is selected by the algorithm using the destination system port value. In the netstat -an command output, the TCP/IP address suffix in the Foreign column is the TCP or UDP destination port value.
round-robin	default	Outgoing traffic is spread evenly across all the adapter ports in the EtherChannel. This mode is the typical choice for two hosts connected back-to-back (without an intervening switch).

Round-Robin

All outgoing traffic is spread evenly across all of the adapters in the EtherChannel. It provides the highest bandwidth optimization for the AIX server system. While round-robin distribution is the ideal way to utilize all the links equally, consider that it also introduces the potential for out-of-order packets at the receiving system.
In general, round-robin mode is ideal for back-to-back connections running jumbo frames. In this environment, there is no intervening switch, so there is no chance that processing at the switch could alter the packet delivery time, order, or adapter path. On this direct cable network path, packets are received exactly as sent. Jumbo frames (9000 byte MTU) always yield better file transfer performance than traditional 1500 byte MTUs. In this case, however, they add another benefit. These larger packets take longer to send so it is less likely that the receiving host would be continuously interrupted with out-of-order packets.
Round-robin mode can be implemented in other environments but at increased risk of out-of-order packets at the receiving system. This risk is particularly high when there are few, long-lived, streaming TCP connections. When there are many such connections between a host pair, packets from different connections could be intermingled, thereby decreasing the chance of packets for the same connection arriving out-of-order. Check for out-of-order packet statistics in the tcp section of the netstat -s command output. A steadily-increasing value indicates a potential problem in traffic sent from an EtherChannel.
If out-of-order packets are a problem on a system that must use traditional Ethernet MTUs and must connected through a switch, try the various hash modes offered in standard mode operation. Each mode has a particular strength, but the default and src_dst_port modes are the logical starting points as they are more widely applicable.

Standard or 8032ad

Standard algorithm. The standard algorithm is used for both standard and IEEE 802.3ad-style link aggregations. AIX divides the last byte of the "numerical value" by the number of adapters in the EtherChannel and uses the remainder to identify the outgoing link. If the remainder is zero, the first adapter in the EtherChannel is selected; a remainder of one means the second adapter is selected, and so on (the adapters are selected in the order they are listed in the adapter_names attribute).
The Hash Mode selection determines the numerical value used in the calculation. By default, the last byte of the destination IP address or MAC address is used in the calculation, but the source and destination TCP or UDP port values may also be used. These alternatives allow you to fine-tune the distribution of outgoing traffic across the real adapters in the EtherChannel.
In default hash mode, the adapter selection algorithm is applied to the last byte of the destination IP address for IP traffic. For ARP and other non-IP traffic, the same formula is applied on the last byte of the destination MAC address. Unless there is an adapter failure which causes a failover, all traffic between a host pair in default standard mode goes out over the same adapter. The default hash mode may be ideal when the local host establishes connections to many different IP addresses.
If the local host establishes lengthy connections to few IP addresses, however, you will notice that some adapters carry a greater load than others, because all the traffic sent to a specific destination is sent over the same adapter. While this prevents packets from arriving out-of-order, it may not utilize bandwidth in the most effective fashion in all cases. The port-based hash modes still send packets in order, but they allow packets belonging to different UDP or TCP connections, even if they are sent to the same destination, to be sent over different adapters, thus utilizing better the bandwidth of all the adapters.
In src_dst_port hash mode, the TCP or UDP source and destination port values of the outgoing packet are added, then divided by two. The resultant whole number (no decimals) is plugged into the standard algorithm. TCP or UDP traffic is sent on the adapter selected by the standard algorithm and selected hash mode value. Non-TCP or UDP traffic will fall back to the default hash mode, meaning the last byte of either the destination IP address or MAC address. The src_dst_port hash mode option considers both the source and the destination TCP or UDP port values. In this mode, all of the packets in one TCP or UDP connection are sent over a single adapter so they are guaranteed to arrive in order, but the traffic is still spread out because connections (even to the same host) may be sent over different adapters. The results of this hash mode are not skewed by the connection establishment direction because it uses both the source and destination TCP or UDP port values.
In src_port hash mode, the source TCP or UDP port value of the outgoing packet is used. In dst_port hash mode, the destination TCP or UDP port value of the outgoing packet is used. Use the src_port or dst_port hash mode options if port values change from one connection to another and if the src_dst_port option is not yielding a desirable distribution.

Managing EtherChannel and IEEE 802.3ad Link Aggregation

This section will tell you how to perform the following tasks:

• Listing EtherChannels or Link Aggregations
• Changing the Alternate Address
• Adding, removing, or changing adapters in an EtherChannel or Link Aggregation
• Remove an EtherChannel or Link Aggregation
• Configure or remove a backup adapter on an existing EtherChannel or Link Aggregation

Listing EtherChannels or Link Aggregations

1. On the command line, type smit etherchannel.
2. Select List All EtherChannels / Link Aggregations and press Enter.

Changing the Alternate Address

This enables you to specify a MAC address for your EtherChannel or Link Aggregation.

1. On AIX 5.2 with 5200-01 and earlier, type ifconfig interface detach, where interface is your EtherChannel's or Link Aggregation's interface. (On AIX 5L with 5200-03 and later, you can change the alternate address of the EtherChannel without detaching its interface).
2. On the command line, type smit etherchannel.
3. Select Change / Show Characteristics of an EtherChannel and press Enter.
4. If you have multiple EtherChannels, select the EtherChannel for which you want to create an alternate address.
5. Change the value in Enable Alternate EtherChannel Address to yes.
6. Enter the alternate address in the Alternate EtherChannel Address field. The address must start with 0x and be a 12-digit hexadecimal address (for example, 0x001122334455).
7. Press Enter to complete the process.
   Note:
   Changing the EtherChannel's MAC address at runtime may cause a temporary loss of connectivity. This is because the adapters need to be reset so they learn of their new hardware address, and some adapters take a few seconds to be initialized.

Dynamic Adapter Membership

Prior to AIX 5L with 5200-03, in order to add or remove an adapter from an EtherChannel, its interface first had to be detached, temporarily interrupting all user traffic. To overcome this limitation, Dynamic Adapter Membership (DAM) was added in AIX 5L with 5200-03. It allows adapters to be added or removed from an EtherChannel without having to disrupt any user connections. A backup adapter can also be added or removed; an EtherChannel can be initially created without a backup adapter, and one can be added a later date if the need arises
Not only can adapters be added or removed without disrupting user connections, it is also possible to modify most of the EtherChannel attributes at runtime. For example, you may begin using the "ping" feature of Network Interface Backup while the EtherChannel is in use, or change the remote host being pinged at any point.
You may also turn a regular EtherChannel into an IEEE 802.3ad Link Aggregation (or vice versa), allowing users to experiment with this feature without having to remove and recreate the EtherChannel.
Furthermore, with DAM, you may choose to create a one-adapter EtherChannel. A one-adapter EtherChannel behaves exactly like a regular adapter; however, should this adapter ever fail, it would be possible to replace it at runtime without ever losing connectivity. To accomplish this, you would add a temporary adapter to the EtherChannel, remove the defective adapter from the EtherChannel, replace the defective adapter with a working one using Hot Plug, add the new adapter to the EtherChannel, and then remove the temporary adapter. During this process you would never notice a loss in connectivity. If the adapter had been working as a standalone adapter, however, it would have had to be detached before being removed using Hot Plug, and during that time any traffic going over it would simply have been lost.

Adding, removing, or changing adapters in an EtherChannel or Link Aggregation

There are two ways to add, remove, or change an adapter in an EtherChannel or Link Aggregation. One method requires the EtherChannel or Link Aggregation interface to be detached, while the other does not (using Dynamic Adapter Membership, which is available in AIX 5L with 5200-03 and later).

Making changes to an EtherChannel using Dynamic Adapter Membership

Making changes using Dynamic Adapter Membership does not require you to stop all traffic going over the EtherChannel by detaching its interface. Consider the following before proceeding:

Notes:

1. When adding an adapter at runtime, note that different Ethernet adapters support different capabilities (for example, the ability to do checksum offload, to use private segments, to do large send, and so forth). If different types of adapters are used in the same EtherChannel, the capabilities reported to the interface layer are those supported by all the adapters (for example, if all but one adapter supports the use of private segments, the EtherChannel will state it does not support private segments; if all adapters do support large send, the channel will state it supports large send). When adding an adapter to an EtherChannel at runtime, be sure that it supports at least the same capabilities as the other adapters already in the EtherChannel. If you attempt to add an adapter that does not support all the capabilities the EtherChannel supports, the addition will fail. Note, however, that if the EtherChannel's interface is detached, you may add any adapter (regardless of which capabilities it supports), and when the interface is reactivated the EtherChannel will recalculate which capabilities it supports based on the new list of adapters.
2. If you are not using an alternate address and you plan to delete the adapter whose MAC address was used for the EtherChannel (the MAC address used for the EtherChannel is "owned" by one of the adapters), the EtherChannel will use the MAC address of the next adapter available (in other words, the one that becomes the first adapter after the deletion, or the backup adapter in case all main adapters are deleted). For example, if an EtherChannel has main adapters ent0 and ent1 and backup adapter ent2, it will use by default ent0's MAC address (it is then said that ent0 "owns" the MAC address). If ent0 is deleted, the EtherChannel will then use ent1's MAC address. If ent1 is then deleted, the EtherChannel will use ent2's MAC address. If ent0 were later re-added to the EtherChannel, it will continue to use ent2's MAC address because ent2 is now the owner of the MAC address. If ent2 were then deleted from the EtherChannel, it would start using ent0's MAC address again. Deleting the adapter whose MAC address was used for the EtherChannel may cause a temporary loss of connectivity, because all the adapters in the EtherChannel need to be reset so they learn of their new hardware address. Some adapters take a few seconds to be initialized.
   If your EtherChannel is using an alternate address (a MAC address you specified), it will keep using this MAC address regardless of which adapters are added or deleted. Furthermore, it means that there will be no temporary loss of connectivity when adding or deleting adapters because none of the adapters "owns" the EtherChannel's MAC address.
3. Almost all EtherChannel attributes can now be modified at runtime. The only exception is Enable Gigabit Ethernet Jumbo Frames. To modify the Enable Gigabit Ethernet Jumbo Frames attribute, you must first detach the EtherChannel's interface before attempting to modify this value.
4. For any attribute that cannot be changed at runtime (currently, only Enable Gigabit Ethernet Jumbo Frames), there is a field called Apply change to DATABASE only. If this attribute is set to yes, it is possible to change, at runtime, the value of an attribute that usually cannot be modified at runtime. With the Apply change to DATABASE only field set to yes the attribute will only be changed in the ODM and will not be reflected in the running EtherChannel until it is reloaded into memory (by detaching its interface, using rmdev -l EtherChannel_device and then mkdev -l EtherChannel_device commands), or until the machine is rebooted. This is a convenient way of making sure that the attribute is modified the next time the machine boots, without having to disrupt the running EtherChannel.

To make changes to the EtherChannel or Link Aggregation using Dynamic Adapter Membership, follow these steps:

1. At the command line, type smit etherchannel.
2. Select Change / Show Characteristics of an EtherChannel / Link Aggregation.
3. Select the EtherChannel or Link Aggregation that you want to modify.
4. Fill in the required fields according to the following guidelines:
   • In the Add adapter or Remove adapter field, select the Ethernet adapter you want to add or remove.
   • In the Add backup adapter or Remove backup adapter fields, select the Ethernet adapter you want to start or stop using as a backup.
   • Almost all the EtherChannel attributes may be modified at runtime, although the Enable Gigabit Ethernet Jumbo Frames attribute cannot.
   • To turn a regular EtherChannel into an IEEE 802.3ad Link Aggregation, change the Mode attribute to 8023ad. To turn an IEEE 802.3ad Link Aggregation into an EtherChannel, change the Mode attribute to standard or round_robin.
5. Fill in the necessary data, and press Enter.

Making changes on AIX 5.2 with 5200-01 and earlier

Follow these steps to detach the interface before making changes:

1. Type ifconfig interface detach, where interface is your EtherChannel's interface.
2. On the command line type, smit etherchannel.
3. Select Change / Show Characteristics of an EtherChannel / Link Aggregation and press Enter.
4. Select the EtherChannel or Link Aggregation that you want to modify.
5. Modify the attributes you want to change in your EtherChannel or Link Aggregation and press Enter.
6. Fill in the necessary fields and press Enter.

Remove an EtherChannel or Link Aggregation

1. Type ifconfig interface detach, where interface is your EtherChannel's interface.
2. On the command line type smit etherchannel.
3. Select Remove an EtherChannel / and press Enter.
4. Select the EtherChannel that you want to remove and press Enter.

Configure or remove a backup adapter on an existing EtherChannel or Link Aggregation

The following procedure configures or removes a backup adapter on an EtherChannel or Link Aggregation. This option is available only in AIX 5.2 and later.

1. Type ifconfig interface detach, where interface is your EtherChannel's or Link Aggregation's interface.
2. On the command line, type smit etherchannel.
3. Select Change / Show Characteristics of an EtherChannel / Link Aggregation.
4. Select the EtherChannel or Link Aggregation that you are adding or modifying the backup adapter on.
5. Enter the adapter that you want to use as your backup adapter in the Backup Adapter field, or select NONE if you wish to stop using the backup adapter.

Troubleshooting EtherChannel

If you are having trouble with your EtherChannel, consider the following:

Tracing EtherChannel

Use tcpdump and iptrace to troubleshoot the EtherChannel. The trace hook id for the transmission packets is 2FA and for other events is 2FB. You cannot trace receive packets on the EtherChannel as a whole, but you can trace each adapter's receive trace hooks.

Viewing EtherChannel Statistics

Use the entstat command to get the aggregate statistics of all the adapters in the EtherChannel. For example, entstat ent3 will display the aggregate statistics of ent3. Adding the -d flag will also display the statistics of each adapter individually. For example, typing entstat -d ent3 will show you the aggregate statistics of the EtherChannel as well as the statistics of each individual adapter in the EtherChannel.

Note:

In the General Statistics section, the number shown in Adapter Reset Count is the number of failovers. In EtherChannel backup, coming back to the main EtherChannel from the backup adapter is not counted as a failover. Only failing over from the main channel to the backup is counted. In the Number of Adapters field, the backup adapter is counted in the number displayed.

Improving Slow Failover

If the failover time when you are using network interface backup mode or EtherChannel backup is slow, verify that your switch is not running the Spanning Tree Protocol (STP). When the switch detects a change in its mapping of switch port to MAC address, it runs the spanning tree algorithm to see if there are any loops in the network. Network Interface Backup and EtherChannel backup may cause a change in the port to MAC address mapping.
Switch ports have a forwarding delay counter that determines how soon after initialization each port should begin forwarding or sending packets. For this reason, when the main channel is re-enabled, there is a delay before the connection is re-established, whereas the failover to the backup adapter is faster. Check the forwarding delay counter on your switch and make it as small as possible so that coming back to the main channel occurs as fast as possible.
For the EtherChannel backup function to work correctly, the forwarding delay counter must not be more than 10 seconds, or coming back to the main EtherChannel might not work correctly. Setting the forwarding delay counter to the lowest value allowed by the switch is recommended.

Adapters not Failing Over

If adapter failures are not triggering failovers and you are running AIX 5.2 with 5200-01 or earlier, check to see if your adapter card needs to have link polling enabled to detect link failure. Some adapters cannot automatically detect their link status. To detect this condition, these adapters must enable a link polling mechanism that starts a timer that periodically verifies the status of the link. Link polling is disabled by default. For EtherChannel to work correctly with these adapters, however, the link polling mechanism must be enabled on each adapter before the EtherChannel is created. If you are running AIX 5L with 5200-03 and later, the link polling is started automatically and this cannot be an issue.
Adapters that have a link polling mechanism have an ODM attribute called poll_link, which must be set to yes for the link polling to be enabled. Before creating the EtherChannel, use the following command on every adapter to be included in the channel:

smit chgenet

Change the Enable Link Polling value to yes and press Enter.

Using Jumbo Frames

For the jumbo frames option to work properly in AIX 5.2 and earlier, aside from enabling the use_jumbo_frame attribute on the EtherChannel, you must also enable jumbo frames on each adapter before creating the EtherChannel using the following command:

smitty chgenet

Change the Enable Jumbo Frames value to yes and press Enter. On AIX 5.2 and later, jumbo frames are enabled automatically in every underlying adapter when it is set to yes.

Remote Dump

Remote dump is not supported over an EtherChannel.

IEEE 802.3ad Link Aggregation

IEEE 802.3ad is a standard way of doing link aggregation. Conceptually, it works the same as EtherChannel in that several Ethernet adapters are aggregated into a single virtual adapter, providing greater bandwidth and protection against failures. For example, ent0 and ent1 can be aggregated into an IEEE 802.3ad Link Aggregation called ent3; interface en3 would then be configured with an IP address. The system considers these aggregated adapters as one adapter. Therefore, IP is configured over them as over any Ethernet adapter.
Like EtherChannel, IEEE 802.3ad requires support in the switch. Unlike EtherChannel, however, the switch does not need to be configured manually to know which ports belong to the same aggregation.
The advantages of using IEEE 802.3ad Link Aggregation instead of EtherChannel are that it creates the link aggregations in the switch automatically, and that it allows you to use switches that support the IEEE 802.3ad standard but do not support EtherChannel.
In IEEE 802.3ad, the Link Aggregation Control Protocol (LACP) automatically tells the switch which ports should be aggregated. When an IEEE 802.3ad aggregation is configured, Link Aggregation Control Protocol Data Units (LACPDUs) are exchanged between the server machine and the switch. LACP will let the switch know that the adapters configured in the aggregation should be considered as one on the switch without further user intervention.
Although the IEEE 802.3ad specification does not allow the user to choose which adapters are aggregated, the AIX implementation does allow the user to select the adapters. According to the specification, the LACP determines, completely on its own, which adapters should be aggregated together (by making link aggregations of all adapters with similar link speeds and duplexity settings). This prevents you from deciding which adapters should be used standalone and which ones should be aggregated together. The AIX implementation gives you control over how the adapters are used, and it never creates link aggregations arbitrarily.
To be able to aggregate adapters (meaning that the switch will allow them to belong to the same aggregation) they must be of the same line speed (for example, all 100 Mbps, or all 1 Gbps) and they must all be full duplex. If you attempt to place adapters of different line speeds or different duplex modes, the creation of the aggregation on the AIX system will succeed, but the switch may not aggregate the adapters together. If the switch does not successfully aggregate the adapters together, you may notice a decrease in network performance. For information on how to determine whether an aggregation on a switch has succeeded, see Troubleshooting IEEE 802.3ad.
According to the IEEE 802.3ad specification, packets going to the same IP address are all sent over the same adapter. Thus, when operating in 8023ad mode, the packets will always be distributed in the standard fashion, never in a round-robin fashion.
The backup adapter feature is available for IEEE 802.3ad Link Aggregations just as it is for EtherChannel. The backup adapter does not need to be connected to an IEEE 802.3ad-enabled switch, but if it is, the backup adapter will still follow the IEEE 802.3ad LACP.
You can also configure an IEEE 802.3ad Link Aggregation if the switch supports EtherChannel but not IEEE 802.3ad. In that case, you would have to manually configure the ports as an EtherChannel on the switch (just as if a regular EtherChannel had been created). By setting the mode to 8023ad, the aggregation will work with EtherChannel-enabled as well as IEEE 802.3ad-enabled switches. For more information about interoperability, see Interoperability Scenarios.

Note:

The steps to enable the use of IEEE 802.3ad varies from switch to switch. You should consult the documentation for your switch to determine what initial steps, if any, must be performed to enable LACP in the switch.

For information in how to configure an IEEE 802.3ad aggregation, see Configuring IEEE 802.3ad Link Aggregation.

Considerations

Consider the following before configuring an IEEE 802.3ad Link Aggregation:

• Although not officially supported, the AIX implementation of IEEE 802.3ad will allow the Link Aggregation to contain adapters of different line speeds; however, you should only aggregate adapters that are set to the same line speed and are set to full duplex. This will help avoid potential problems configuring the Link Aggregation on the switch. Refer to your switch's documentation for more information on what types of aggregations your switch allows.
• If you will be using 10/100 Ethernet adapters in the Link Aggregation on AIX 5.2 with 5200-01 and earlier, you need to enable link polling on those adapters before you add them to the aggregation. Type smitty chgenet at the command line. Change the Enable Link Polling value to yes, and press Enter. Do this for every 10/100 Ethernet adapter that you will be adding to your Link Aggregation.
  Note:
  In AIX 5L with 5200-03 and later, enabling the link polling mechanism is not necessary. The link poller will be started automatically.

Configuring IEEE 802.3ad Link Aggregation

Follow these steps to configure an IEEE 802.3ad Link Aggregation:

1. Type smit etherchannel at the command line.
2. Select Add an EtherChannel / Link Aggregation from the list and press Enter.
3. Select the primary Ethernet adapters that you want on your Link Aggregation and press Enter. If you are planning to use a backup adapter, do not select the adapter that you plan to use for the backup at this point. The backup adapter option is available in AIX 5.2 and later.
   Note:
   The Available Network Adapters displays all Ethernet adapters. If you select an Ethernet adapter that is already being used (has an interface defined), you will get an error message. You first need to detach these interfaces if you want to use them.
4. Enter the information in the fields according to the following guidelines:
   • EtherChannel / Link Aggregation Adapters: You should see all primary adapters that you are using in your Link Aggregation. You selected these adapters in the previous step.
   • Enable Alternate Address: This field is optional. Setting this to yes will enable you to specify a MAC address that you want the Link Aggregation to use. If you set this option to no, the Link Aggregation will use the MAC address of the first adapter.
   • Alternate Address: If you set Enable Alternate Address to yes, specify the MAC address that you want to use here. The address you specify must start with 0x and be a 12-digit hexadecimal address (for example, 0x001122334455).
   • Enable Gigabit Ethernet Jumbo Frames: This field is optional. In order to use this, your switch must support jumbo frames. This will only work with a Standard Ethernet (en) interface, not an IEEE 802.3 (et) interface. Set this to yes if you want to enable it.
   • Mode: Enter 8023ad.
   • Hash Mode: You can choose from the following hash modes, which will determine which data value will be used by the algorithm to determine the outgoing adapter:
     • default: In this hash mode the destination IP address of the packet will be used to determine the outgoing adapter. For non-IP traffic (such as ARP), the last byte of the destination MAC address is used to do the calculation. This mode will guarantee packets are sent out over the EtherChannel in the order they were received, but it may not make full use of the bandwidth.
     • src_port: In this hash mode the source UDP or TCP port value of the packet will be used to determine the outgoing adapter. If the packet is not UDP or TCP traffic, the last byte of the destination IP address will be used. If the packet is not IP traffic, the last byte of the destination MAC address will be used.
     • dst_port: In this hash mode the destination UDP or TCP port value of the packet will be used to determine the outgoing adapter. If the packet is not UDP or TCP traffic, the last byte of the destination IP will be used. If the packet is not IP traffic, the last byte of the destination MAC address will be used.
     • src_dst_port: In this hash mode both the source and destination UDP or TCP port values of the packet will be used to determine the outgoing adapter (specifically, the source and destination ports are added and then divided by two before being fed into the algorithm). If the packet is not UDP or TCP traffic, the last byte of the destination IP will be used. If the packet is not IP traffic, the last byte of the destination MAC address will be used. This mode can give good packet distribution in most situations, both for clients and servers.
     To learn more about packet distribution and load balancing, see Load-balancing options.
   • Backup Adapter: This field is optional. Enter the adapter that you want to use as your backup. The backup adapter option is available in AIX 5.2 and later.
   • Internet Address to Ping: This field is optional, and only available if you have only one adapter in the main aggregation and a backup adapter. The Link Aggregation will ping the IP address or host name that you specify here. If the Link Aggregation is unable to ping this address for the Number of Retries times in Retry Timeout intervals, the Link Aggregation will switch adapters.
   • Number of Retries: Enter the number of ping response failures that are allowed before the Link Aggregation switches adapters. The default is three. This field is optional and valid only if you have set an Internet Address to Ping.
   • Retry Timeout: Enter the number of seconds between the times when the Link Aggregation will ping the Internet Address to Ping. The default is one second. This field is optional and valid only if you have set an Internet Address to Ping.
5. Press Enter after changing the desired fields to create the Link Aggregation.
6. Configure IP over the newly-created Link Aggregation device by typing smit chinet at the command line.
7. Select your new Link Aggregation interface from the list.
8. Fill in all the required fields and press Enter.

Managing IEEE 802.3ad

For management tasks that can be performed on an IEEE 802.3ad Link Aggregation after configuration, see Managing EtherChannel and IEEE 802.3ad Link Aggregation.

Troubleshooting IEEE 802.3ad

If you are having trouble with your IEEE 802.3ad Link Aggregation, use the following command to verify the mode of operation of the Link Aggregation:

entstat -d device

where device is the Link Aggregation device.

This will also make a best-effort determination of the status of the progress of LACP based on the LACPDUs received from the switch. The following status values are possible:

• Inactive: LACP has not been initiated. This is the status when a Link Aggregation has not yet been configured, either because it has not yet been assigned an IP address or because its interface has been detached.
• Negotiating: LACP is in progress, but the switch has not yet aggregated the adapters. If the Link Aggregation remains on this status for longer than one minute, verify that the switch is correctly configured. For instance, you should verify that LACP is enabled on the ports.
• Aggregated: LACP has succeeded and the switch has aggregated the adapters together.
• Failed: LACP has failed. Some possible causes are that the adapters in the aggregation are set to different line speeds or duplex modes or that they are plugged into different switches. Verify the adapters' configuration. In addition, some switches allow only contiguous ports to be aggregated and may have a limitation on the number of adapters that can be aggregated. Consult the switch documentation to determine any limitations that the switch may have, then verify the switch configuration.

Note:

The Link Aggregation status is a diagnostic value and does not affect the AIX side of the configuration. This status value was derived using a best-effort attempt. To debug any aggregation problems, it is best to verify the switch's configuration.

Interoperability Scenarios

The following table shows several interoperability scenarios. Consider these scenarios when configuring your EtherChannel or IEEE 802.3ad Link Aggregation. Additional explanation of each scenario is given after the table.

Table 5. Different AIX and switch configuration combinations and the results each combination will produce.

EtherChannel mode	Switch configuration	Result
8023ad	IEEE 802.3ad LACP	OK - AIX initiates LACPDUs, which triggers an IEEE 802.3ad Link Aggregation on the switch.
standard or round_robin	EtherChannel	OK - Results in traditional EtherChannel behavior.
8023ad	EtherChannel	OK - Results in traditional EtherChannel behavior. AIX initiates LACPDUs, but the switch ignores them.
standard or round_robin	IEEE 802.3ad LACP	Undesirable - Switch cannot aggregate. The result may be poor performance as the switch moves the MAC address between switch ports

• 8023ad with IEEE 802.3ad LACP: This is the most common IEEE 802.3ad configuration. The switch can be set to passive or active LACP.
• standard or round_robin with EtherChannel: This is the most common EtherChannel configuration.
• 8023ad with EtherChannel: In this case, AIX will send LACPDUs, but they will go unanswered because the switch is operating as an EtherChannel. However, it will work because the switch will still treat those ports as a single link.
  
  Note:
  In this case, the entstat -d command will always report the aggregation is in the Negotiating state.
• standard or round_robin with IEEE 802.3ad LACP: This setup is invalid. If the switch is using LACP to create an aggregation, the aggregation will never happen because AIX will never reply to LACPDUs. For this to work correctly, 8023ad should be the mode set on AIX.

For more information visit :- http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.commadmn/doc/commadmndita/etherchannel_intro.htm