For a major trans-oceanic backbone provider, at least 15ish years ago they had a mile or two between Detroit and Chicago where both ends were on the same side of the interstate highway.
But it's more frequent on DAS (Distributed Antenna Systems, AKA small-cell or micro-cell) networks.
Also the challenge of when fibers are leased (if that's still a thing, based on the networks I helped design I'd say 'probably').
They really are analogous to Lamport's "Distributed System" quip; A damaged fiber owned by a company you have never heard of can wreck your day.
In route one circuit from Washington to London via ashburn and down to Texas before going on subsea circuits through mainland Europe to ensure path diversity. The other heads north east towards Newark and and surfaces in Bude. That took two years of contractural arguments.
Fibres on the Texas link die all the time (yearly) but that’s fine.
My circuits in the far east are often carried one via dues and one via the USA.
It is possible to guarentee diversity, you just need to get specific detail and have good contractual protections.
If you design your network badly sure. Spend 3 months with BR once trying to get diverse routing out of one location to their two nearby exchanges, took a long time for them to work out what they needed to do as they had the two paths crossing in a single location for quite some time, but eventually they sorted it.
lol there’s supposed to be …. But then greed gets in the way lol.
Usually it’s a combo of a tier 1 that decides that a resell agreement is “good enough” and “well just eat the loss” and then by the time stuff like this rolls around “we’ll get back to you” + a bunch of silly explanations that boil down to “you’re not gonna sue us though” start coming out lol
Nest door neighbor did this when he just decided on a whim to put in a new driveway and started digging with a bobcat. 10 mins in and BLAM severed the main Comcast coax serving the entire neighborhood that was running under his driveway.
Luckily we were on Century Link so weren't affected by his stupidity. lol
Years ago I was touring a POP in a small city. They were very proud to point out one fiber coming in one side of the building going South and another on the opposite side of the building going North.
This is a pretty classic network operator story: go to great lengths to provide for physically diverse paths, then not notice when your provider refactors and grooms them onto the same bundle.
I find intonation it’s better to select two competing provides as they are less likely to use the same ducts.
Had a few issues in some lamdlpcked counties on occasion - Kabul had all traffic routing via the same route into Pakistan for a few years, Kathmandu had an earthquake knock out both ISPs within 30 seconds of each other.
Backups via starlink global roaming work well now in particularly challenging environments.
In NL this often happens because of waterways. All starts off with good intentions, multiple providers, totally different fibers exiting the building in different directions. And then they have to cross a canal or a river. It's a 50/50 at that point whether they converge on the same conduit across the water.
Over ten years ago, my employer was spinning up a new DC across the state and had three links between it and the primary DC. Two were pretty direct, but we needed a third because at one point in the 300 mile path, the two main links went within 400 meters of each other.
So how national-security-adjacent critical systems like an airport system doesn't have a larger set of backup links, and validates that they are geographically separate up until the connections, and have realtime alerting on the connection status, is surprising to me.
> validates that they are geographically separate up until the connections
This isn't relevant in this case. The problems occurred at different places. The backup fiber was, separately from the issues with the primary system, cut by a construction crew. It's not two fiber lines both cut at the same spot.
You can get two separate failures in short succession, where the first one hasn’t been fixed. For critical devices that’s why you have tertiary backups, ideally on different technology compel you. For one major event I had diverse fibre and satelite and microwave. Lost microwave and satelite to electronic warfare but the fibres were ok. Ofcom were really pissed about it, but nothing they could do.
> An RFTS enables fiber to be automatically tested from a central location. A central computer is used to control the operation of OTDR-like test components located at key points in the fiber network. The test components scan the fiber to locate problems. If a problem is found, its location is noted and the appropriate personnel are notified to begin the repair process.
That is fascinating. Out of curiosity, how far apart would the paths have had to remain in order to be ok with just two links? Like, I'm thinking a plane crash between those two links in the 400m zone could destory both links so a third link makes sense, but if it were 800m that would probably be fine? Or what magnitude of disaster are they trying to mitigate and how much separation would that require for just two links?
Similar events happened in Europe disguised as thieves stealing fiber optic. This does not have any sense economically, as the value in the market is zero so... either is an honest accident and is cleared in a few days, or is sabotage
>disguised as thieves stealing fiber optic. This does not have any sense economically, as the value in the market is zero
Do you honestly think that crackheads think that far in advance?
I've seen fiberoptic cables stolen from 2 (city) jobsites in the last 5 years, once by tweakers later caught trying to sell them as scrap copper and the second thief was never caught.
This happened even though the spools had big signs on them saying "Fiber Optic Cable - NO COPPER".
This used to happen more often in the past, when copper thieves mistook fiber for conductor. There were cases where critical systems were shut down due to fiber optic theft, and the thieves were caught burning the sheathing to expose the... glass.
That's very uncharacteristic, especially given all those contingency requirements (backup policies, failovers, testing schedules etc.) imposed on corporations/companies in the wake of 9/11.
You actually have to ensure that someone isn't just checking boxes the tests passed and the tests actually passed.
I worked for a regional ISP that had a major outage when the redundant fiber provided by the telephone company was cut in one place triggering a full loss of connectivity. It also caused a massive 911 outage for 200,000 people as it isolated the 911 center from the city core.
Turns out the phone company didn't connect one side of the ring topology even though they certified they did. Needless to say lawsuits abounded.
And auditing that is expensive and you always get given hell from the assholes going “why am I paying for this if nothing has gone wrong”
The amount of shit I was given everytime I went through a checklist working at a fedramp certified company working with emergency alerts on the phone system made me prematurely gray.
I could feel the barely contained seething rage everytime I told the execs that the reason this release will take 3 days and not be instantaneous like your friends releases at a faang and that information made you embarrassed at your dinner party, is because you agreed to this process contractually years ago and now it’s a crime if I just sign off on it being ok without actually checking that it’s ok.
Yea, fangers are def not used to working on life critical systems. I personally have avoided it as much as possible, but not completely, for all the reasons you listed above. No need for me to accept that much responsibility for other peoples lives when I've been able to earn a living just fine not doing so.
There are plenty of people who are happy to check off "Backups" boxes, or talk about their backup strategy, or whatever - while hoping the day never comes when they actually need that tricky "Restore" feature.
Glacier, and I think a couple of Iron Mountain products, exist for "cheap backups, really expensive restores" because the restore is going to be paid for by insurance (or by a counterparty when it's explicitly used for escrow.) So there is an explicit market for this (in the backup-of-things, rather than backup-of-capabilities, space.)
You don't always have a copy of your hardware to restore onto. And the test's entire purpose is that you're not yet sure whether your restore will truly work. So you can't just run a backup and restore on your true prod system, because you're not sure it won't wreck it. So you need extra money to have a second system onto which you try to restore. If you don't have a lot of money, you will want to actually use your disks for storage, not to put them into a second testing server. Of course I'm not talking about very professional companies with super critical data. Just simpler smaller scale places or consumers.
No, even professional companies with critical data balk at this.
I worked at a company worth a few billion and the leadership balked when they told engineering they wanted a near instantaneous failover system and our department informed them that would require paying for a second environment that could be rolled over to.
It is rare to find leaders who can accept the cost of redundant infrastructure that is there for emergency backup.
What puzzles me is why they can’t accept it when they are perfectly fine with insurance costs and I can’t see much of a difference between the two when looking at a spreadsheet of costs other than possibly tax differences between the type of expenditure.
Sure, that's another category and different considerations. I've worked at an academic lab with a limited budget where we set up file servers, but couldn't afford to do anything approximating 3-2-1. We did a nightly backup of a tiny part (most important) of the data onto another server in a different building, but like 90% of the data was just YOLO (well, RAID, but that's not a backup), and that's just how it is. Disks are pretty good though, they don't die often nowadays, and when people accidentally deleted their data, it was just gone. Would have been cool to have a backup of everything, but even just pulling out a nightly backup from a dense server with dozens of terabytes isn't simple and you don't want to slow down the server by constantly reading just for constructing the backups. It's a tradeoff.
Redundancy has costs and those costs can be spent elsewhere like having higher quality or bigger disks, or a faster network switch or better CPUs etc.
In theory, it would also be better to own two cars instead of one, because what if the first one gets in an accident or just breaks down. Yet, not everyone can afford that. Should you just buy two half-as-expensive cars than what you can buy one of, so you can say you have a "backup"? Likely the two half-price ones would be so much crappier that the one good car would cause you less trouble in expectation than driving a shitty one and then having another shitty spare one, both of which will constantly have issues.
I've worked on backup/failure systems since the mid-90s, and I've found there's one universal truth: If you don't fully test your backup/failure system, you don't have a backup/failure system.
There's generally two wrong responses: (1) We spent a lot of money on 'blah blah blah', a lot of other companies use it, so yeah, we've got a backup/failure system. And, (2) inadequate testing - either, we tested 1 of 50 services, and it worked, so the whole system can be restored; or, we gracefully tested, and it worked, so it will obviously work during not-graceful incidents.
And the root cause of this is generally that no one gets promoted for implementing an adequate backup/failure system, or it's extremely rare.
Besides what the others have said, speaking from a neteng perspective, sometimes backup lines (and the core infrastructure in general) are engineered in such a way that it can't easily be tested properly without taking other things down, or manually rolling a truck specifically to test it in isolation with extra equipment.
Not saying that's what is going on here, just that it's possible.
While probably it's legit because I've seen Dave discussing the process on the YT channel (which I no longer watch) he unfortunately does have form https://news.ycombinator.com/item?id=39813625
Having TMOG on a .org when it sells commercial licences doesn't sit well with me given the history.
I'm a little confused by your comment.. especially considering the fact that you edited it and changed the link from and old comment thread from 2024 to whatever this dead post is from 2012. What does having form mean? Who is Dave?
> You didn’t really have one big app that you could open and order it to entertain you. Instead, you had a few dozen of bookmarks to websites, each with a specific idea in mind.
This overlooks how common custom homepages such as Lycos, Yahoo! and MSN were and how the browsers pushed these as their default home pages.
While they may not be as addictive as "the algorithm" of today, they were still packed with clickbait images/links to attract your attention and ads to drain your wallet.
At no point in time has anyone ever suspected that companies requiring arbitration was in the best interests of the consumer. This is very plainly an attempt to avoid responsibility and consequences. It's a margin-improving policy. The safety of its customers are merely a number on a spreadsheet.
The MP3 lawsuits were filed in the districts in which the defendants resided. Almost all of the 94 districts had MP3 sharing lawsuits.
You are probably thinking of patents. For a long time the Eastern District of Texas (EDTX) was by far the most popular choice for patent lawsuits by far, especially against big tech companies.
Then there was a Supreme Court decision (TC Heartland LLC v. Kraft Foods Group Brands LLC, 581 U.S. 258 (2017)) [1]. It made it so corporations can only be sued for patent infringement in the state where they are incorporated or a district where they have both infringed and have a regular and established place of business.
EDTX is largely a rural district with no really major cities. Most big tech companies have no physical offices there.
Many of those cases moved to the Western District of Texas (WDTX). WDTX includes Austin, and a whole lot of big tech has offices there.
There was one judge in Waco whose court was particularly attractive to patent lawsuits, due to having plaintiff-friendly procedures and a rapid trial schedule. 100% of patent cases filed in Waco went to him. In a few years his court was handling 20% of all patents cases in the entire country.
The Supreme Court and Congress did not like so many patent cases being handled by a single judge, and the WDTX Chief Judge making it so patent cases filed in WDTX are random distributed among its 13 judges.
Cases against big tech companies mostly moved to Delaware (where many big tech companies are incorporated) or Northern California.
EDTX regained the number one spot for patent cases after that, although not from big tech cases. Delaware and Northern California remain the big places for those.
EDTX is #1 because even though it is rural and has almost no big tech presence, it still has major retailers, grocery chains, distribution centers, and communication hubs. For patent suits not against big tech it still works fine.
Also, Heartland did not apply to international defendants. They can still be sued in any district in the US.
The biggest reason EDTX is a popular choice if you can pick it is not plaintiffs do particularly well there. It depends on when you look because it has varied quite a bit, but there are often other districts where plaintiffs win more.
The big reasons are:
• Patent litigation is complex, both for the court and the attorneys. Regardless of which side of the case is on you want a court with experience. If the court is inexperienced the case can drag on which is expensive, burning up lots of billable hours of the expensive patent litigation firm you are using. EDTX has many patent experienced judges, and has developed rules and procedures to move cases along.
• In addition to cases moving along efficiently once they get going, the get going faster in EDTX.
Patent suits are civil cases. They take a back seat to criminal cases. EDTX simply does not have much federal crime (or state crime that ends up in federal district courts).
I can't say I agree with forcing your team to use your own personal "framework" of approaching a problem or else you'll get "feedback".
> Sometimes I will give feedback to people when they are missing steps in the framework, or are poorly executing some of the steps. I expect the same feedback in return.
I also don't like that urgency is built in as the standard process either, no wonder everyone is burnt out.
Acting with urgency is a bit at odds with discovering flaws in your plan. If you're sprinting you're less likely to notice smells and things that are inelegant, more likely to paper them over. That said, there is a time for urgency. Just not every single task.
> I can't say I agree with forcing your team to use your own personal "framework" of approaching a problem
Is that not prt of what leadership always is? It might be less explicit. People often don't write their process down. Things get more confusing because people don't understand what is being asked of them.
But when is there ever no process that has to be followed? What leader lets people do whatever however and what's the role of that leader?
It is called micromanagement when the boss tells you _how_ you should think. My approach is when leading a tech team to find together the things which should be done and find a method together to keep us on the path. Facilitation. Forcing decisions to be made. Keep quality high enough. Let people find what they enjoy doing and is useful. This is really basic and well known set of tools.
Well, for a torrent to stay healthy, users have to seed after download, so it can't possibly have the same UX as a standard browser file download unless you want to either kill the ecosystem or hide from the user what is consuming upload bandwidth.
That said, for large files, I much prefer the UX of a well-designed torrent client like Transmission to my web browser. If nothing else, the downloads are reliably resumable.
> I much prefer the UX of a well-designed torrent client like Transmission to my web browser. If nothing else, the downloads are reliably resumable.
Brave browser had BitTorrent client built in for a while. I tried it a couple of times as I already use Brave for web browsing on my laptop. It was a very confusing BitTorrent client. I struggled to use it, and wasted time waiting for a download to complete only to not be able to find where the files were and then they disappeared. Using a decent BitTorrent client like you say is much preferable to the one that they had in Brave browser.
Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.
I wonder how long it was down? Days, weeks, months?
reply