Basic Network Design Principles for Electrical Engineers
Substation design, equipment sizing, protection settings configuration, communication network design… wait, what!?
The field of power system protection, automation, and control is dramatically changing with technology and innovation. Electromechanical relays were very popular in the past but since the introduction of microprocessor-based and numerical relays, electromechanical relays are facing obsolescence. Most relays today, or rather commonly known to young protection engineers as IEDs (Intelligent Electronic Devices) are multifunction. This means that aside from the typical protection functions, these IEDs can do metering, disturbance and event recording all in the same box. Setting a relay has transformed from the basic dials into basic programming, complex but not complicated.
Protection engineers have to adapt to this changing landscape by learning new skill sets. While we are busy learning and ‘programming’ our IEDs for a green- or brown-field project, a new transformation is looming around the corner and it’s HUGE! Here comes the introduction of a communications-based standard for power system protection, automation, and control, the IEC 61850. It’s communications-based, and most probably, not a strong point for protection engineers.
We may see ourselves knowledgeable when it comes to short-circuit, phasors, damage curves, overcurrent, differential, distance protection but topics or subjects involving network design, packets, frames, protocol, TCP/IP, MAC, VLAN and everything about communication networks can be alien to us. This article aims to discuss the basic networking design principles necessary in our understanding and application of IEC 61850. The topics will be discussed in a level of detail so as to not complicate the understanding of the subject.
Data Networks Classification by Spatial Scope
Network design starts with knowing the classification of networks. These networks are classified into various categories but the most common is by spatial scope.
- Personal Area Network (PAN) – very short distance. An example is Bluetooth.
- Local Area Network (LAN) – connects hosts in a single office or building. It can be as simple as two PCs and a printer in someone’s home office. Examples are Ethernet and Wireless LAN (WLAN).
- Metropolitan Area Network (MAN) – stretched over an urban area. Example: WiMAX
- Wide Area Network (WAN) – extends over a large area. Spanning a town, province, or country. If LAN interconnects hosts, WAN interconnects switches, routers, or modems. It is often run by communication companies. Example: Mobile Broadband
Protocol Layering
Protocols are very basic in communication networks and understanding them is fundamental in network design. They define the rules in order to actually communicate between devices. Communication among devices within a network can be quite complex hence the introduction of Protocol Layering. Layering allows the division of a complex task into several simpler tasks that facilitate communication between devices. Consider for example the following scenario.
A couple in a long-distance relationship needs to communicate with one another. However, because of the distance, a face-to-face conversation is not possible. To be able to communicate, they will need to use airmail services. The process starts with one writing the letter. The letter is then encrypted to add privacy before it is sent through airmail. The recipient on the other end will receive the letter via airmail, decrypt it before being able to read the contents of the letter. Sounds simple right?
This is how protocol layering works. In our example, the task of sending a message between hosts is divided into three simpler tasks, write/read, encrypt/decrypt, and send/receive. These are called layers. In protocol layering, two principles should be observed,
First is that each layer should perform two opposite tasks. In our example, these tasks are write/read, encrypt/decrypt, and send/receive.
Second, each layer input and output at both sites should be the same. In our example, the different tasks are the same at both ends.
OSI and TCP/IP
One cannot start network design without knowledge of the OSI. The Open Systems Interconnection (OSI) was created as a reference model to promote the interoperability of products from different vendors. It is composed of 7 layers. On the other hand, the Transmission Control Protocol/Internet Protocol (TCP/IP) was created based on OSI. TCP/IP is widely used in different applications today.
Instead of 7 layers from the OSI model, TCP/IP integrated the Presentation, and Session layer into the Application layer effectively making TCP/IP operate on 5 layers.
- Application Layer – responsible for process-to-process communication. Examples: HTTP, SMTP, FTP, TELNET.
- Transport Layer – provides services to the application layer. Flow control, error control, congestion control services are available. Examples: Transmission Control Protocol (TCP) – connection oriented and User Datagram Protocol (UDP) – connectionless.
- Network Layer – defines end-to-end delivery of packets (from source to destination). No flow control, no error control, no congestion control. Example: Internet Protocol (IP) – connectionless.
- Data Link Layer – responsible for moving the packet through individual links by encapsulating it in a frame.
- Physical Layer – transmission medium; includes analog/digital signals (bits)
Consider the example shown in figure 7. A source host from one network connects to a destination host on another network. From the source computer, the data is encapsulated through the TCP/IP layers before sending it out to the switch. Encapsulation occurs at the transport layer where a TCP header is added on the data coming from the application layer resulting in what is called a segment for TCP or user-datagram for UDP. The segment/datagram is passed on to the network layer, takes this as payload, encapsulates it by adding the IP header resulting in what is called a packet. On the link layer, the packet is taken as a payload and encapsulated by adding its own header resulting in what is called a frame. The frame is then sent through the physical layer and on to the next hop, the switch.
The switch operates only up to layer two, thus no encapsulation/decapsulation occurs. The switch forwards the data to the router. The router, on the other hand, operates up to layer three. This time, decapsulation occurs at the link layer by removing the link layer header. The link layer passes the packet to the network layer and then to the link layer where encapsulation occurs. The frame is then passed on through the physical layer for transmission. Another switch, layer 2, no encapsulation/decapsulation occurs. At the destination host, decapsulation occurs starting from the link layer up to the transport layer. The headers are removed at each layer until the data reaches the application layer.
Data Link Layer
We move our discussion on network design to the data link layer since this is where GOOSE and SV messages run. Under this are two sublayers, the Logical Link Control (LLC) and the Media Access Control (MAC). Our interest will be the MAC. We will leave the readings about LLC to you.
Media Access Control is sometimes called link-layer address or physical address or link address or MAC address. These terms are used interchangeably. MAC address is a 48-bit unique identifier burned into each and every Ethernet network interface in a hexadecimal format. The first three octets identify the organization or the vendor to which the device belongs. This is called the Organizationally Unique Identifier (OUI) and is provided and maintained by IEEE. On the other hand, the last three octets are network interface controller specific and are assigned by the organization or vendor.
Consider the following MAC address, 00:60:78:00:FD:0B. The first three octets 00:60:78 tells us that this device is from Power Measurement, Ltd. In the same sense, IEC TC-57, an organization behind IEC 61850, has an OUI of 00:0C:CD. Thus, MAC addresses used in IEC 61850 should have the OUI of IEC TC-57.
It is also important to note that the first byte of the MAC address may change depending on how it was used or configured (though it refers to the same OUI). The least and second-least significant bit of this byte gives provides more information about the MAC address in that from these bits, we can determine if the address is a multicast or a unicast address, and whether the address was universally or locally administered.
The Ethernet
Network design in the link-layer usually deals with Ethernet. Ethernet is the most widely used method in connecting between hosts on the same network. It operates on the link layer of the TCP/IP. This is why IEC 61850 was designed to work on Ethernet. A format of a typical Ethernet frame is shown in the figure below.
Source Address
The MAC address of the sending station is identified as the source address of the Ethernet frame. This address is the MAC address of the source device (computer, relay, etc.).
Destination Address
Unlike the source address, the destination address is not necessarily the MAC address of the destination device. It can be the MAC address of the end device or an address representing specific devices or an address of all the devices on the network.
- Unicast – if the message is intended for a single recipient in the network, the destination address is the MAC address of the destination device.
- Multicast – if the message is intended for several recipients in the network, the destination address is not the MAC address of all or any of the recipients. Instead, the destination address is a multicast MAC address. This address can be set by setting the least significant bit of the first byte to be 1. In IEC 61850, GOOSE messages can have a multicast MAC address from 01:0C:CD:01:00:00 up to 01:0C:CD:01:01:FF while SV messages can have a multicast MAC address from 01:0C:CD:04:00:00 up to 01:0C:CD:04:01:FF.
- Broadcast – if the message is intended to all hosts in the network, the destination address is a MAC address in the form of FF:FF:FF:FF:FF:FF.
Multicast Caveat!
Multicast behaves like broadcast. If no recipient groups are specified in the switch, it sends the packet to all ports!
Managing Network Traffic
A quite challenging task in network design is managing your traffic.Multicast frames behave like broadcast frames. Consider for example the figure below. Multicast frames from all devices A, B, C, D, E, and F will be received by all of them. For device E, it will receive the multicast frames from device A, B, C, D, and F. The same happens to all devices, which significantly decreases the available network bandwidth and devices’ processing power.
To remedy the issue, multicast filtering is applied in the switch. In the above example, if device E is connected to port 6 of switch 2 and it only needs the messages from A, C, and F, a MAC filter is set-up in switch 2 to allow only the specific MAC addresses to pass through port 6.
| Port | Destination MAC |
| 6 | 01:0C:CD:04:00:01 |
| 01:0C:CD:04:00:02 | |
| 01:0C:CD:04:00:03 |
In the illustration below, only frames from A, C, and F will reach E.
Ethernet switches will not forward frames to the same recipient at the same time to avoid collisions. For example, if the switch simultaneously receives frames from A, B, C, D which are all addressed to B, only 1 frame can be sent at a time while others are buffered. If this was in a substation automation environment, IEC 61850 for example, a protection trip GOOSE message will have to wait before it can be processed by the switch.
To allow prioritization of specific messages, VLANs can be set-up in the network. With VLAN, multicast frames are limited to their broadcast domain defined by VLAN. VLAN allows the prioritization of frames. As an example, GOOSE trip message should be assigned with higher priority compared to GOOSE messages containing the temperature of the transformer.
There are eight (8) priority levels available, 1-lowest(background), 7-highest(network control), 0-higher than 1 (default). Consider the figure below, if the message from device D has a higher priority than all other messages in the network, it will be prioritized by the switches in the queue.
Redundancy in Switched Networks
Another challenge in network design is increasing the reliability of your network. Switched networks can be set-up to allow back-up or redundancy in case of failure of any of the links. The basic failure recovery options are the Spanning Tree Algorithms. These are:
- Spanning Tree Protocol
- Rapid Spanning Tree Protocol
- Multiple Spanning Tree Protocol
Figure 14a. STP Network 
Figure 14b. STP Failover
Though STAs achieve failure recovery, the duration of this ranges from 5-100ms. In the application of IEC 61850 for substation automation, this failover time is not sufficient because SV sampling interval is in the range of 208µs and 250µs for 60Hz and 50Hz systems, respectively.
In order to ensure the reliability of the substation automation system, IEC 61850 recommends the use of Parallel Redundancy Protocol (PRP) or High-Availability Seamless Redundancy (HSR) to achieve zero failover time.
Parallel Redundancy Protocol (PRP)
PRP achieves redundancy by duplicating the incoming data packets and transmitting them into two separate and independent LANs. The packet which reaches the destination host first is accepted while the duplicate packet is discarded. PRP works with Ethernet and is completely transparent. The key takeaway is that the PRP identifier is part of the frame payload, and is ignored by switches. The disadvantage to PRP is that the two LANs should be completely independent of each other, they cannot be connected or bridged together in any way.
High-availability Seamless Redundancy (HSR)
In HSR, all devices are connected to the network in a ring configuration. The destination host of the redundant copies of the HSR frame accepts the first copy of the message and discards the second one. The source host accepts massages around the ring except for the message that it sent. Unlike PRP, HSR tag replaces part of the frame header information which makes the Ethernet frame unidentifiable by devices that are not HSR capable. These devices can only be connected to an HSR ring via a Redundancy Box.
Summary
- Knowledge on network design principles is important in substation automation systems.
- GOOSE operates on a Local Area Network (LAN).
- Protocols are placed to facilitate host-to-host communication.
- TCP/IP is used on the internet today. It is based on the OSI Model.
- Data Link layer as defined by the Ethernet Standards.
- Media Access Control (MAC) facilitates the movement of frames within the link layer through a source/destination address.
- Link-layer communication can be unicast, multicast, or broadcast.
- Unicast is point-to-point.
- Multicast has several recipients.
- Broadcast is sent to all nodes.
- GOOSE and SV messages are multicast frames.
- Multicast behaves like broadcast if not managed.
- VLAN segregation and Multicast filtering are two methods to manage multicast frames.
- Reliability is now dependent on network conditions.
- Spanning Tree Algorithms enable network healing but in a range of 5-100 milliseconds.
- PRP and HSR are recommended for IEC 61850 because of the ‘zero’ switch-over period.
References
Forouzan, B. (2013). Data Communications and Networking, Fifth Edition. New York, NY: McGraw Hill.
MAC Address. Retrieved from https://en.wikipedia.org/wiki/MAC_address
Student Guide – Switching in Industrial Networks with RUGGEDCOM. RUGGEDCOM Educational Services
GOOSE Principles , Networking Architecture & Workshop. IEC 61850 University
Check out other articles on our website.
Keep going, today you gained a new subscriber ) )
Thank you! We’re surely at it.
This is a very detailed explanation on how data is moving over different types of network.
Thank you for sharing your thoughts with the us.