Introduction
Actors:
$dude - DevOps hired by the client company.
$colleague - My colleague, stuck in the same quagmire as I am.
To be short, we are an internet provider. We are not IT support. Nothing of the following mentioned in the post is in our job description. Nor is it part of the clients contract or part of an SLA. As an ISP our responsibilities end at the end of the fiber-optic strand. Unless the customers router is owned by us or a contractual obligation or an SLA say otherwise.
However, in the past, when we were a lot smaller and thus had a lot more free time, we used to give a helping hand to our customers pro bono, even for things that were not our direct responsibility. But now we are trying to fend that off when possible. It is ok if it is like 5 mails a year(single mails, not mail chains) and maybe a single visit to the customer a year and it turns out to not be our fault or responsibility. We tolerate that much. But not more than that. And of course this is only if it was not our mistake. When only a single customer has an issue, it tends to not be one of our mistake. When you make a mistake in an ISP it is usually at least the whole block that goes down :), And of course we have other more important work to do, like making sure our WAN network works as expected and has enough capacity to carry increasing amounts of traffic.
The plot thickens
But now, we have an old client from many years ago. Back from the time when we did those 'pro bono favours. We did those more to court clients rather than having any real responsibility to do them. And that was OK. It was not too much, a couple of mails here and there, a couple of minutes of work.
All until they hired their new DevOps $dude. That DevOps came from another company that was also our client, but with that client we have mistakenly signed an SLA that was really unfavourable for us. We still haven't gotten over the mistake that SLA was. But that is a story for another day.
However, spoiler, this story does indeed have a good ending. Even before all of this we have already learned the best way to 'educate' such clients. But more about that later.
Principally, $dude was used to from his last job that we had to be obedient soldiers to him. My $colleague even explicitly told him that he is now in a new company with a very different business relationship to us. He said 'ok' but in reality he did not care at all about that. Go in one ear and out the other. His stance both in his old and new job was 'not my job'. And the only thing I have seen the guy do was to scourge external contractors(I am kidding a bit, but his former company had that disgusting outsource everything mentality).
$dude is a little obsessed with replacing every router he sees with IDS/IPS firewalls. Which is not bad but... A little known thing about ISPs is that ISPs don't really do much with firewalls if anything at all. Mostly because the amount of traffic going trough an ISP makes hardware firewalls prohibitively expensive and impractical. Hardware firewalls capable of doing and IPS filtering traffic over multiple 100G interfaces cost about as much as real estate where I live. Therefore ISPs work with routers, not firewalls, the only firewall features on those routers usually being just access-lists. Over these 6 years working at an ISP there was barely a handful of times when I have been configuring a firewall with IPS/IDS capabilities.
And this time we were smart. We have predicted that tickets are gonna start flowing hard and fast, a lot of tickets. We did not lease the firewall as part of our service. We sold the firewall to them. Because we have sold that firewall, it is their device. We do not have a responsibility to support it. They accepted the offer and bought the device. Although we still have a user account on the device, but so do they. We are willing to delete our user account whenever.
Months passed, tickets came, a lot of ball passing over emails. 2-3 client visits to them.
When one day $dude calls my $colleague. When there are problematic cases like these, $colleague prefers to put them on the speaker phone so that everyone can chime in and understands the situation.
$dude: Hello, this is $dude from company X. There is no wireless in a part of our office, we can barely reach speeds of 20-30 Mbps. Can you please come over to us?
$colleague: Can you please tell me what is the colour of the lights on the affected AP?
$dude: It is white.(Which means the AP has no access to the firewall which also acts as the APs controller).
$dude: I see that the patch cable is damaged.
$colleague: Have you tried to physically reboot the AP and change the patch cable?
$dude(starts shouting): I DON"T GIVE A DAMN. I AM THE CLIENT. THAT IS YOUR JOB. THE PATCH CABLE IS OVERRUN, DAMAGED...
$colleague(visibly irritated, tuns off his microphone): Yes you are a client, but a client of what? Which of our services are you a client off? IT support is not in your package because we don't even do that in the first place.
$colleague(turns on his microphone):OK, someone will come to take a look. Goodbye.
As soon as the call ended, $colleague went to the secretary and asked her nicely if she can tell him what service does the client have contracted with us. The secretary told him: Only an internet service, nothing more. I don't remember the exact speed. Some 200M or 300M symmetric, nothing much. And my colleague had a huge grin on his face.
At the same time I have logged into the Firewall/AP controller and I have confirmed, out of 10 APs one was inaccessible, and another was meshed(it was not supposed to be meshed). Everything else was normal. A problem exists, but alas it was not under our 'jurisdiction'.
We send two of our new interns/technicians out to the client. And we explicitly tell them to fill out the work order in two copies and round out their time to the hour. As an ISP, none of this was our equipment, job or responsibility. We are an ISP NOC.
And we charged them for that :)
The good ending
2 technicians x 1 hour. Around here the market rate for a technician-hour is around 100$. So that ended up costing them 200$ just to replace two patch cables. And they had their own IT guy $dude who even identified the damaged cables, but was unwilling to do anything about it himself.
That is the good ending. Even earlier we have learned that the best way to wean off clients from 'little favours' when it becomes too much... is to simply start charging the favours.
Even better, later $colleague proactively noticed that the firewall needed a firmware upgrade and contacted the client about it. The firmware upgrade was really needed. we did not plan to charge them for that.
However $dude, the first thing he asks is: Will you also charge us for that?
We did charge them.
A little bit unrelated, but why do many DevOps, not all of them, but a big part of them... Why do they hate the 'ops' side of their work? I don't get it. On one side you have boring programming, clunky automation and flaky cloud providers, while on the other side you have interesting networks, sexy servers, innovative on premes infrastructure...
No comments:
Post a Comment