The Challenge
I recently faced the challenge of building a virtual chassis (VC) with two EX4650 switches. The task itself was not the hard part, of course. What made it difficult was the fact that VCs on EX4650 switches only work with 40G or 100G connections, and the only QSFP+ module we had that could support the distance between the two switches had an MTP connector, while the patch panels connecting the two buildings had ST connectors.
The (maybe) Solution
The only solution I could come up with was to use breakout cables to go from MTP to 4x LC, then use LC connectors to go from LC to ST and finally connect to the patch panel. I know it sounds ridiculous, but hey, being a nerd, I love to experiment with over-engineered setups.
So the next logical step for me was to build this setup in our lab and try to build a VC across 4x10G breakouts.
When I tried to research if it was even possible to build a VC over breakouts, I couldn't find any resources to help me. Perhaps this was due to my poor ability to write good search queries (I'm pretty sure I'm not the first person in the world to try and build a VC over breakouts), but it looked like I was going to have to try and find out for myself.
The Lab
There were some limitations I couldn't overcome. For example, I couldn't replicate the exact setup with the ST patch panels. Also, I couldn't place the two lab nodes 160m apart, which is roughly the distance between the two bespoken switches to connect them. But I tried to get as close as possible to the real setup.
So I used LC to LC connectors to connect two EX4650 via MTP to 4x LC breakout cables on both sides. Here is a picture of what it looked like:
When I connected the two switches using this cable setup, I noticed that the EX4650 automatically channeleized ports where the appropriate QSFP modules and breakout cables were used:
# Switch 1
root@SW1> show interfaces terse | match xe-0
xe-0/0/48:0 up up
xe-0/0/48:0.16386 up up
xe-0/0/48:1 up up
xe-0/0/48:1.16386 up up
xe-0/0/48:2 up up
xe-0/0/48:2.16386 up up
xe-0/0/48:3 up up
xe-0/0/48:3.16386 up up# Switch 2
root@SW2> show interfaces terse | match xe-0
xe-0/0/48:0 up up
xe-0/0/48:0.16386 up up
xe-0/0/48:1 up up
xe-0/0/48:1.16386 up up
xe-0/0/48:2 up up
xe-0/0/48:2.16386 up up
xe-0/0/48:3 up up
xe-0/0/48:3.16386 up upOk, so the good news is that connecting the LC connectors together with these adapters actually works and all the lanes are up - not a big surprise as this is basically what a fibre patch panel does.
The next question is, does running a VC over this connection work? The Juniper documentation says the following about building a VC with EX4650 switches:
EX4650 switches do not have dedicated or default-configured VCPs, but you can set any of the 40-Gigabit Ethernet QSFP+ or 100-Gigabit Ethernet QSFP28 uplink ports on the front panel (non-channelized ports 48 through 55) as VCPs.
They emphasise that the ports should not be channelized...
But I assume that Juniper means that we can't use one lane of the breakout to build a VC and use the other lanes for normal network traffic (at least I hope that's what they mean, lol...). Since I was going to connect all four lanes together, I figured it's worth a try.
So I decided to simply convert port 48 on both switches to VC ports after pre-provisioning the VC:
# Same actions and outputs on both switches
root@SW1> show configuration virtual-chassis
preprovisioned;
no-split-detection;
member 0 {
role routing-engine;
serial-number XH37230*****;
}
member 1 {
role routing-engine;
serial-number XH37230*****;
}# Same actions and outputs on both switches
root@SW1> request virtual-chassis vc-port set pic-slot 0 port 48
Port conversion initiated, use show virtual-chassis vc-port to verifyNow the prize question - did it work???
Look at this output:
root@SW1> show virtual-chassis
Preprovisioned Virtual Chassis
Virtual Chassis ID: a359.14d4.ac97
Virtual Chassis Mode: Enabled
Mstr Mixed Route Neighbor List
Member ID Status Serial No Model prio Role Mode Mode ID Interface
0 (FPC 0) Prsnt XH37230***** ex4650-48y-8c 129 Master* N VC 1 vcp-255/0/48
1 (FPC 1) Prsnt XH37230***** ex4650-48y-8c 129 Backup N VC 0 vcp-255/0/48It really did, amazing!!
Some Tests and the Results
Although it looked like this setup would work fine, I didn't want to celebrate just yet, having had some bad experiences in the past when I was happy to build a VC with two EX4650 switches over 10G interfaces. JunOS didn't complain, and at first it looked like it would work. But after connecting a couple of access switches to the VC, some bad side effects occurred and suddenly the whole FPC on both members was behaving erratically.
So I wanted to run a few tests before declaring the setup a success.
I connected an access switch over an ether channel with LACP enabled, first with two connections to one of the VC members. After running it for a while and sending some traffic from the access switch to the VC, I could see promising results. All LACP interfaces were up, collecting and distributing. I also didn't experience any problems similar to the scenario with the VC built over 10G links. I then connected a link to each of the members and sent some traffic over the ether channel again. The results here were positive as well.
Finally, I did some failover tests. It turns out that even if only one of the four lanes has a problem, the VC will fail. All lanes need to be up for the VC to be stable. So in terms of risk, this setup introduces four times more potential points of failure.
A disclaimer at the end - this is obviously not the optimal way to build a VC and this is just one possible solution for this exact scenario. You should always stick to Juniper validated network designs to be on the safe side.
In the end, I think we can consider this lab setup a success. To be honest, I expected to run into problems with this setup, as I couldn't find any resources on the internet and it didn't seem like it should be a valid option. But I am even more pleased with the positive outcome.
Comments