Sunday 25 September 2022

Oh Mikrotik...

    Mikrotik is a network equipment manufacturer, a relative newcommer to the field. They are known for making feature rich cheap devices that are popular in the small business, small ISP and prosumer markets.

     I have now several years of experience managing several hundreds of Mikrotik devices. I have used them as Core routers, CPE routers, core switches, distribution switches, access switches. All of that in an internet service provider context.So I think I am somewhat qualified of judging Mikrotik equipment. Luckily we migrated away from MIkrotik in core routing and any Mikrotik switching. 

    In essence Mikrotiks Routeros which runs (almost) all Mikrotik devices is a far far removed fork of Vyattta OS, which means that somewhere down there is a Linux kernel. However, there is no way that I am aware of to reach the underlying Linux system.

    Indeed I will now share my love/hate relationship with Mikrotik.In some cases these cheap devices are surprisingly good in others, surprisingly bad.

The good

Routing

     Mikrotik has many good sides in its routing. It has also many bad sides in its routing but more on that later. The good first. All Mikrotik devices support all of the widely used routing protocols. BGP, OSPF, RIP, MPLS, IPv4 and IPv6... And they are not just bare-bones implementations, you get BFD, all the little features you expect and even more. Some of those features are not supported by larger and more expensive network equipment manufacturers. And the best part is that you get all thesame features on a 30$ Mikrotik that you get on the 3000$ Mikrotik(with some very minor exceptions. Naturally I do not recommend abusing the 30$ device with tasks it is not capable of doing.

CPE router

    As a CPE, device it supports every feature imaginable. Minus maybe an analog phone adapter, but come on, it is 2022. It supports about 15 or so types of VPN tunnels, the more notable being site to site IPSEC, wireguard, zerotier, EoIP, MPLS, OpenVPN...

    Naturally it supports PPPoE, it supports VRRP. Some models have integrated LTE modems, so 3G/4G/5G is supported as well. It's stability is beyond what is expected of consumer devices. Some models have integrated Wifi. All models support traffic shaping and traffic queuing.

Features

    I have mentioned that RouterOS supports many many features. Well it really is a swiss army knife of various ethernet computer networking functionalities.

Torrent client - MikroTik

     Let me talk about some of the more impressive features it supports that I have not mentioned already. Routing first. VRFs, yes, you can get VRFs on a 30$ device. Mesh routing support. Packet marking for routing. VPLS and traffic engineering. IGMP multicast routing. PIM multicast routing.

    Now switching... MLAG on select models. VXLAN support. QinQ VLAN support. Layer 2 filtering. Layer 2 NAT.

    As for the rest... You can use it as a radius server. You can use the integrated hotspot server. . You can use it as a web proxy. You can use it even as a TFTP, FTP or SMB file server. Therre is also LORA support for some IoT devices etc...

    You can install Docker on ARM based Mikrotik models. That means that theoretically you can use it as anything. You can run Doom on Mikrotik if you want to.

    These are only some of the more surprising features that Mikrotik supports, there are many others.

Stability 

    Some Mikrotik devices are quite stable. You won't experience many reboots with  some devices. I have noticed that it all depends on the CPU architecture used. MIPSBE  devices can have uptimes of several years if their power supply was not interrupted.

Price

    As previously mentioned, Mikrotik devices tend to be rather cheap. Of course, even Mikrotik has cheaper home use devices and their flagship datacenter oriented devices which are a bit pricier. On one end you can buy a 23$ Mikrotik HAP mini and you have the CCR 1072  at 3350$. In between them you have the CCR2216, which is cheaper than the 1072, but newer and better.

The Dude and Winbox

    Winbox is the best way to manage Mikrotik routers. Forget the CLI. Usually I prefer the CLI on other devices, but Mikrotik is an exception. Winbox is really well polished, it exposes 99% of everything that is in the CLI and has some features tof its own that are not present in the CLI. PLus there are some drawbacks of the CLI that I will mention later.

    The Dude is my favourite network mapping tool. It supports Mikrotik devices as well as it has good support for 3rd party devices. It is best used for mapping on the layer 2 level. It is fairly easy to use, drag and drop, and then you just configure the SNMP data and any usernames and passwords.

Home use

   Considering the price of these devices. It is barely a little more expensive than the average consumer device. So it is a premium device for home users, prosumers and even homelabbers. You can get a 10G Mikrotik RB 4011 for 200$, and before 2022 inflation I remember it was going for about 170$.

Mikrotiks NAT implementation

    Mikrotiks NAT implementation is super flexible. You can basically change the entire packet header how you want it to be changed.

The Bad

Switching

    Uh, switching on Mikrotik. Let's say I have had bad experiences with Mikrotik switching, and luckily we have removed Mikrotik switches from all important areas.

    0) First off Mikrotik has a really blurry line between switches and routers. Most Mikrotik "switches" come with "RouterOS". Very few Mikrotik "switches" come with "SwitchOS". Some have dual boot, but you will almost certainly just use RouterOS. Most Mikrotik switches are in fact layer3 switches that are even as layer 3 switches closer to routers than to proper layer 2 switches. Now, there is nothing wrong with layer 3 switches, but the problem with Mikrotik is that there is no clear distinction between switches and routers. Now it was a couple of years ago since we attempted to use Mikrotik switches. So they have released newer and more powerful models and things might have been fixed. But I have found better alternatives.

    Switch OS is really barebones  and has lamost no featuress at all. It barely supports VLANs. There is not even a CLI, just a web interface to configure it. Thus I will provide no more attention to Switch OS as I rarely ever use it.

    Be aware, what I will tell you was tested on their best and most expensive 10G switches at the time(2020).

    1) One issue  that we experienced with Mikrotik switches were frequent packet drops. About 1 in every 10000 packets were lost  between two mikrotik switches. With fiber terminationss that tested as perfect, with multiple changed pairs of SFP+ modules. WIth clean factory rerset configuration.

    2) Another issue we experienced is that with some switches we would get random port disconnects and reconnects every couple of days. We have replaced  the fiber patches, the fiber cores in the cable, the SFP+ modules, we have upgraded RouterOS. And it were fairly short fiber runs. Only replacing the Mikrotik switch helped.

    3) A third issue is that it is much too easy to misconfigure a Mikrotik switch in such a way that you will not get hardware line speed. In fact the factory default configuration is not configured for hardware line speed. With other vendor switches you have to go out of your way to configure a switch in such a way that hardware line speed will be broken. With Mikrotik it is a minefield and you have to spend hours on the wiki just for that. On top of it all this differs for many different Mikrotik models, so you have to be very careful.

    4) A fourth issue is the clusterfuck that is VLAN  configuration on MIkrotiks. There is at least 5 different ways to configure VLAN trunking on Mikrotik devices. And it all depends on which Mikrotik model you have. Some of those don't work on some models, others do, and you have to figure it out. I have found one way that works on all but it is very stupid and problematic. You essentially make VLAN subinterfaces for VLANs and you bridge them together. BUT AT LEAST IT WORKS ON EVERY MIKROTIK.

Core routing

    We did use Mikrotik as a core router. Specifically we used the CCR 1036 and the CCR 1072 which were Mikrotiks falgship routers back then. It was an improvement over our old software router. But alas we outgrew Mikrotik  in less than a year since we implemented it. It was tolerable when our routing table was like 5 full  BGP tables. But our routing table grew to  20+ full BGP tables. And our core of Mikrotiks just had big problems with that.

    1) The first fault is that for performance routing Mikrotik does not use Layer 3 ASICs. Correction, now there is the CCR 2216 which has a layer 3 ASIC, but back then that was nonexistant.

    2) The second and worse fact was that in RouterOS v6 the entire routing process was single threaded. Yes, the entire routing process was stuck on a single CPU core. What good are  72 CPU cores on the Mikrotik CCR 1072 if the entire routing process is choking on a single core? And our entire BGP routing table had about 20+ full BGP routing tables inside of it(think of it as if our routing table had 20 internets full of routing tables inside of it). As our routing table grew, eventually it became so bad for us that putting a single static route in our routing table took 30+ minutes after we pressed enter, until the route was finally in the routing table. When a CCR rebooted(which happened as well, but not too often) it took about 4:30 HOURS for a full iBGP propagation to finish.

    3) As for 3, when we tried to implement OSPF as our IGP we would get random OSPF neighbor dissconnects. Probably related to 2, as BGP and OSPF and the entire rest of routing shared the same CPU processs.

    Now, 2 is fixed fixed in RouterOS v7. Probably. I don't know as right when RouterOS v7 came we ditched MIkrotik out of our core routing. RouterOS v7 also entirely rewrote all of routing and used an entirely new syntax that is incompatible with the old syntax. And syntax conversion on upgrades does not always work. So, good that we avoided that landmine. We have now moved to Cisco routing in our core, and we are happy with that and have not looked back.

Consultores da Depressão | Facebook

Wireless

    Wireless on Mikrotik leaves a lot to be desired. First off, Mikrotik wireless is always a couple generations behind. Mikrotik adopted Wifi 5 Wave2 only last year, when their competitors had Wave 2 since 2016-2017. Mikrotik has finally now(september 2022) started  creating its first models with Wifi6 support when their competitors have adopted Wifi6 in 2019/2020. Mikrotiks competitors are now starting with Wifi6e, and talk is that Wifi7 will  be finished in early 2024.

    Now other than lagging in standards support, on Mikrotik wirelesss lans you can expect weak signal coverage, weak troughput and the occasional signal hole. In other words, weak antennas. Their optional central AP management software CAPSMAN, is rudimentary, and weird. At least when compared to their competitor Ubiquiti Unifi, it leaves A LOT to be desired.

    We have also tested their Wireless PTP/PTMP devices a long time ago(around 2015/2016) and they proved no good. The devices sometimes just froze up. Lost connection and the connection would not reestablish until a reboot was done. Singal quality was also mediocre at best. Again pretty bad compared to Ubiquiti Airmax.

IDS/IPS

    Mikrotik has no support for IDS/IPS functionality at all. It does support ACLs, and it has some very rudimentary and very manual setup layer 7 filtering. But that is very far from IDS/IPS support. There are some custom scripts that pretend to actively add  layer 7 filtering, but that is not for serious businesses. Mikrotik has no hardware firewall right now.

Stability 

    Earlier I have stated that depending on CPU microarchitecture  you can have very stable Mikrotik devices. Well also depending on CPU microarchitecture you can have very unstable devices. Sometimes they don't even reboot. Some process gets stuck and stays stuck until you manually reboot the device. I have had the most stability isssues with PowerPC and ARM(not ARM64) based CPUs. It seems that firmware updates have fixed that, as after v6.48 of RouterOS I have not expereienced those issssues, a lot.

    On RouterOS 6.45-6.46 I have process stuck two Mikrotik routers by holding backspace for a minute or two in CLI. Ever since then I avoid Mikrotik CLI and mostly use Winbox. To this day I don't know if this bug was fixed. I still use the Mikrotik CLI occasionally(for example when I need to read the whole config), but my main way of interacting with Mikrotik is WInbox.

Cooling 

    MIkrotik has bought the passive cooling koolaid. That is ok for their more home oriented devices, but thay are putting passive cooling on some(not all) of their datacenter devices. Naturally this proiduces a lot of problems when those devices are insufficiently  cooled. SFP modules start acting funny. The device startss acting funny. And it just gets weird.

Doing too many things 

    MIkrotik has a case of jack of all trades master of none. I love that Mikrotik has a ton of featuress. ut some of those features have a clear lack of attention. They have been programmed once and later forgotten other than the occasional bugfix here and there. Lots of features are implemented but are unstable.

What use are they for? 

CPE router?
    Yes, it is perhaps the best  CPE router for the price/performacen/feature ratio out there right now.

Core router?
    If you are a business smaller than an enterprise. And by enterprise I mean a proper  enterprise with more than 5000 employees. Maybe. An SMB, yes, in fact it is excellent. Mid size business, yes, but be sure to buy their more expensive CCR lineup.Large but not quite enterprise large business, maybe, but buy their best gear if you do so. Enterprise? No.Not for the core network at least.

ISP core router?
    Depends on how large you are. If you are a small ISP and your routing table is 5 or less full BGP routing tables, then yeah, but again, use their CCR 1072,1036 or 2216 routers. And make sure you are on RouterOS v7.

Switches?
    No, at least nothing important.

Wirelesss?
    Maybe. But I would not expect excellent quality out of it. My experience may be outdated though.

Perhaps things have changed? 

    Admittedly  RouterOS v7 is a big game changer. And with RouterOS v7 I have only used MIkrotik as a CPE device. Some of the negatives might have changed, but again, I have found other good vendors that serve those purposes much better. Routing  might have improved a lot, but you know, I have been burned badly once and there is no reason to repeat that. Their switching also might have improved a lot lately, MLAG and VXLAN are not present in v6 but only in v7. I have not tried their PTP/PTMP wireless setups for ages. And they now finally support  wifi6. But still no wifi6 devices with SFP slots :(.

Conclusion

    Mikrtoik makes some good devices. It is good for some uses. Despite its faults. It is very feature rich for the price point. To the point where even I still get surprised by the occasional checkbox or CLI argument.. But for some use cases I would advise against MIkrotik, heavily. Despite the good price. The value lost in customer quality and customer satisfaction can be worse.

    Overall, if you want to use Mikrotik, sure go ahead. Mikrotik can be excellent. I still use it on hundreds of devices. But be aware of MIkrotiks pitfalls.

User story 1: The client has to pay because their IT guy refuses to replace two patch cables.

 Introduction Actors: $dude - DevOps hired by the client company. $colleague - My colleague, stuck in the same quagmire as I am. To be short...