Technology@lemmy.worldbynew_otters_raft@piefed.ca
1 day

CISA Releases Guidance Urging Water Treatment Facilities to Disconnect Equipment From the Internet

www.privacyguides.org English

CISA released an urgent message warning water utilities to disconnect their logic controllers from the internet in the face of rising cyberattacks.

The hacks target internet-facing programmable logic controllers (PLCs) that control equipment and allow machinery to communicate. They monitor and control the water pressure, chemical dosing, and other factors to ensure the water is safe.

Many of the PLCs are apparently open to the internet and use default credentials, allowing a remote attacker to easily take them over.

I assume it is this CISA: https://en.wikipedia.org/wiki/Cybersecurity_and_Infrastructure_Security_Agency

57
    CISA released an urgent message warning water utilities to disconnect their logic controllers from the internet in the face of rising cyberattacks.
    • youmaynotknow@lemmy.zipEnglish
      2 hours

      It’s so fucking dumb. They are getting attacked, and they still need to be told to unplug? What the actual fuck?

    • Doomsider@lemmy.worldEnglish
      7 hours

      I will take things that never should be connected to Internet for $200 Alex.

      • Abyssian@lemmy.worldEnglish
        4 hours

        Right. Why would you do this? Monitoring being available on the Intranet, of course. But controllers on the open internet?

        Might as well really focus on ease of use and just make a live website with one big button that says “cut off all electricity and turn all the water into poison, lol”

          • Abyssian@lemmy.worldEnglish
            2 hours

            And the user support AI the website automatically opens in in popup can just tell you if you tell it you’re the King of Water and you forgot the account info.

      • Duamerthrax@lemmy.worldEnglish
        6 hours

        Remote monitoring? Sure. Remote managing? Fuck that. These facilities shouldn’t be unstaffed for days at a time.

        • BrianTheeBiscuiteer@lemmy.worldEnglish
          5 hours

          It’s called paying people to watch the systems during off-hours. It’s often way cheaper than setting up firewalls, setting up alerts, maintaining patches, paying a VPN company to manage connections, paying another company to handle authentication, and so on. Security constraints are getting so bad at my office I’m seriously thinking of how we can do more things with pen and paper.

        • Doomsider@lemmy.worldEnglish
          5 hours

          Oh for sure, air gapped sensors that relay information are going to be necessary. We should be building these systems with absolute security because they are critical to infrastructure.

          • Duamerthrax@lemmy.worldEnglish
            4 hours

            Some people will say it’s too hard to retrofit onto legacy equipment, but all it really means is adding a secondary system on top of the legacy system. Just a sensor array that has no interaction with existing systems.

            • SanicHegehog@lemmy.worldEnglish
              2 hours

              Yup. Fancy architecture diagram:

                                One Way Radio
                   Transmitter
[Nuclear Reactor] ------------> [Internet connected receiver] -> tHe ClOUd
      • HugeNerd@lemmy.caEnglish
        4 hours

        A simple 300 bps connection over POTS to relay the information from the process controller is all you need. But even POTS is disappearing.

      • nycvin@lemmy.worldEnglish
        6 hours

        Exactly! Whenever I see one of these stories of a utility being hacked that’s my first thought.

    • RepleteLocum@lemmy.blahaj.zoneEnglish
      14 hours

      Not saying I ever accessed it, but there’s a generator in Germany open to the web without security. It makes a few MW. My guess is hospital or any other critical infrastructure.

    • fubarx@lemmy.worldEnglish
      16 hours

      Went on the public tour of our local waste-treatment facility. Really fascinating stuff. Highly recommend it if you’re even slightly curious.

      There are definitely industrial-scale automation systems. Mostly PLCs, but also SCADA for monitoring. Lots of 20-30 year-old tech. Those networks have no reason to be connected to the internet.

      But there were also office equipment and personal devices everywhere. All it takes is for someone wanting to monitor the state of a digester on their laptop from home.

      You just crossed the beams.

      • frank@sopuli.xyzEnglish
        15 hours

        Automation engineer here who does a lot of SCADA

        There are soooo many good ways to securely monitor stuff from home. But especially in critical infrastructure it’s just usually air gapped which is unbeatable in cyber security.

        Tours of any local facility are super cool! I find it all so interesting, seeing how the things we use work!

        • Buckshot@programming.devEnglish
          13 hours

          I worked on new SCADA for a water company across all theirs sites. Previously they were a mismatch of different systems and all airgapped.

          They standardised everything and put them all on VPN. Suddenly you could access the secure water treatment works with 24/7 security guards and the control room was a bunker, from the unmanned rural sewage works with a portacabin for the server, the key hidden under a brick, and wifi for the PLCs.

          The sewage works had always operated like that because it was deemed low risk. No one considered the VPN changed the threat model.

    • sylver_dragon@lemmy.worldEnglish
      1 day

      The hacks target internet-facing programmable logic controllers (PLCs)

      Why the fuck
      is your PLC
      facing the fucking internet!

      Jesus Zombie Christ an a pogo stick. Has no one been paying attention for the last two decades? Seriously, we learned this sort of lesson in Two Thousand and fucking Three. Your critical assets do not get public IP addresses.

      • Godort@lemmy.caEnglish
        1 day

        It usually happens in 2 steps.

        First step is that everything gets connected to the LAN and you can only access the PLC network from within the building’s network. Then some time later, management finds out that keeping someone on-call to go out costs a fortune, so they request that access be made so they can make the change from anywhere.

        The IT team argues security, but no new hardware can be provisioned and a developing a new process is too hard. Then the magical phrase is uttered: “Just make it work”. So IT punches a hole in the firewall, adds a NAT rule, and job done.

        • Ludicrous0251@piefed.zipEnglish
          3 hours

          Except you skipped the part where IT is ridiculously underfunded and treated like crap (“We’re a utility, not a tech company!”)

          With constant turnover, the person receiving the “Just make it work” call is about 6 layers removed from the person who originally set up the infrastructure, and probably completely unaware of all the previous security holes put in “because it’s air-gapped, it doesn’t matter.”

        • sylver_dragon@lemmy.worldEnglish
          1 day

          Sadly yes, this is exactly what happens. And it ends up being IT holding the bag at the end, with the managers having long since cashed out their stock options and left.

          • blargh513@sh.itjust.worksEnglish
            1 day

            “Well, we just bought this new monitoring platform from my cousin’s company and they need access to it. No, their platform doesn’t support any real secure protocols and won’t route over a VPN, it needs to be on the same subnet as the devices. Ok, then just open all ports to the network so they can connect. No, they don’t know what port or protocol, it also runs out of my cousin’s house so it’s on a dynamic IP so you can’t create some type of access rule for it. Just put an any/any rule in and it’s good. What do you mean we already have a monitoring platform? Well, this one is better!”

            • historicaldocuments@lemmy.worldEnglish
              16 hours

              “No, their platform doesn’t support any real secure protocols and won’t route over a VPN, it needs to be on the same subnet as the devices. […]”

              They won’t know that much. They don’t understand that their job isn’t running a water department, it’s running the computers that maintain the water department and that includes security. For someone familiar with computer security it’s easy to recognize negligence, but until the law sees it as criminal negligence nothing’s going to change.

              Small town municipal water departments aren’t going to be able to afford the guy holding the shield against international state actors.

        • Carl@anarchist.nexusEnglish
          1 day

          Literally this. It starts with “everything needs to be on an air gapped LAN so we can control it.” And then suddenly that air gapped LAN isn’t air gapped, because someone got tired of paying the call-out fee every time the on-call person had to drive to the site to push a few buttons on a keyboard. But figuring out a proper VPN (or at least a reverse proxy with access control) was too difficult or expensive, (management thinks the point is to save money on call out fees, not spend money on proper hardware) so they just poked a bunch of port-forwarded holes in the firewall and called it a day.

          • muusemuuse@sh.itjust.worksEnglish
            21 hours

            What’s extra ridiculous is VPNs are not that hard. I am fantastically awful with networking and even I can wrap my head around a WireGuard tunnel.

            • Carl@anarchist.nexusEnglish
              16 hours

              The issue is doing one at scale, with proper access controls. Also, let’s not skip past the whole “network isn’t air gapped anymore” side of things. Now you’re having to maintain firewalls, patch firmwares, manage users, enforce good password hygiene, etc… All without any additional IT support or hardware, because management is doing this as a cost cutting measure.

              I can pretty much guarantee that the systems that get hacked aren’t the ones that do everything right. They’re the ones who have a single shared VPN password, haven’t updated their firewall since it was installed 8 years ago, and they haven’t even changed the default passwords for their control systems. Because they got tired of answering phone calls about what the password was, so they just made every username and password something stupid like admin/admin.

              • Passerby6497@lemmy.worldEnglish
                2 hours

                “network isn’t air gapped anymore” side of things. Now you’re having to maintain firewalls, patch firmwares, manage users, enforce good password hygiene,

                This is all shit they should be doing now. They have to have a firewall to get to the net, so they should be maintaining and patching it. Managing users is standard AD shit, and password hygiene is t as easy, but setting reqs in AD for complexity goes a long way (as does not forcing passwords to rotate on an arbitrary basis).

                And yes, I know it doesn’t happen in a lot of places, which is why I said ‘should’. Having been on the cleanup side of multiple security incidents, doing the basics is harder than you’d expect for a lot of places …

              • muusemuuse@sh.itjust.worksEnglish
                5 hours

                Fine. New rule, anything that’s your fault and costs a company money you get to pay for out of your golden parachute, your pension, your savings, anything. You caused a problem to save money this quarter forsaking the next quarter when you aren’t there? Screw off.

              • sylver_dragon@lemmy.worldEnglish
                10 hours

                Even when a company is doing things pretty well with a VPN, all it takes is one mistake and the cat is out of the bag. At one of my previous employers, we had a fairly good setup with VPNs using single sign on via our IdM provider. IdM enforced SSO and the VPN had a number of policies setup which kept good control over what users could login. And then someone had the brilliant idea to take an old VPN appliance, reset it to factory defaults and configure it just enough to work on our network and then hung it out on the internet for some sort of test (our configuration management left a lot to be desired). Of course, that led to it being left there for a couple months and an attacker finally found it. They also had a valid username/password combination which got them in the door.

                Security is hard and often expensive. But still, compared to “we put out OT devices on the internet” a basic VPN is a huge step up.

      • Dave.@aussie.zoneEnglish
        1 day

        I did some work for a company that owned a /16 public IP range and used it on their internal network. I had to put a safety rated control system on it that ran mining machines that weighed 60 tons and cost a couple of million bucks each.

        They gave me a subnet range for the system that was clearly non-private and I was like, “ummmmm, is that on the Internet?”

        Apparently it wasn’t routed to the rest of the Internet but it always felt like it was one misconfigured router away from disaster.

    • ☂️-@lemmy.mlEnglish
      17 hours

      not so funny when it isn’t happening in the middle east but right at your doorstep

    • Greyghoster@aussie.zoneEnglish
      19 hours

      This has been talked about for years and about how exposed SCADA is. I just assumed that it was fixed/isolated years ago. The fix appears to have been to stop talking about it.

      • VonReposti@feddit.dkEnglish
        16 hours

        That’s why NIS2 in Europe demands that OT is on a separate network from IT. My old workplace (well, the head of FO Technology and his gang anyway) was up in arms about it because it “was a waste of time” (and completely ruin the ideal architecture he had in his mind).

    • ORbituary@lemmy.dbzer0.comEnglish
      1 day

      I am repeating what I said elsewhere on a different comm:

      I call FULL bullshit on this being Iran.

      I personally worked on the cybersecurity incident that happened to St Paul municipal and water systems last year as the Incident Response Technical lead.

      The threat actors were not Iranian then. They were Russian.

      I personally rebuilt most of their networks, VMware hosts, storage, and servers. The majority of the VMware environments did not have config backups and were running 5 and 6 branches of VMware, only two had 7+.

      Their network was insecure. They had IPSEC VPNs between the various location, but there was no further security in that respect. Their firewalls were not being kept up. AAAAND… I warned them that we thought they were still compromised.

      In fact, the forensics team called them out as being clean on a group call one week into the incident, but while we were on the call with leadership, they got RE-infected.

      I had my team compile a massive set of recommendations for them and I wrote the report. Water Systems specifically. They did not listen to us, apparently. The forensics team also did this.

      They had FBI, national guard (fucking incompetent), and law enforcement (useless) in all meetings.

      If they got into Water, they’ll get back into city and municipal again. I guarantee it.

      This is not Iran. It’s the fucking incompetence of city government and Russian threat actors.

      • nymnympseudonym@piefed.socialEnglish
        1 day

        Minor nit: Russia and Iran are already working together on shaheds and targeting

        No surprise at all if they also coordinate cyber

        • ORbituary@lemmy.dbzer0.comEnglish
          17 hours

          Whatever. I worked the fucking case. The infiltrators were Russian. After that, it’s a political or financial negotiation.

      • quarkquasar@lemmy.worldEnglish
        16 hours

        No doubt so trump can make up some bullshit excuse to try to suspend the midterms.

        We already know he’s going to try, I hate how obvious this shit is from a thousand miles away, but still nothing is done about it.

      • historicaldocuments@lemmy.worldEnglish
        16 hours

        If budget weren’t an issue, how many clones of your teams would it take to do that for all the water systems in the US in five years?

        What are the odds that the average muncipality understands the security implications of just existing on the internet these days?

        • ORbituary@lemmy.dbzer0.comEnglish
          6 hours

          10 people per site, per incident, at 50-70 hours/week.

          It’s thankless work and the people in control don’t typically take on contractors. They underpay the actual IR staff. That’s why I left the industry. It’s a big buddy-system network. Really fucked.

          • historicaldocuments@lemmy.worldEnglish
            5 hours

            So to sweep all of the water systems in the country in advance isn’t doable in any time frame?

            IR isn’t an acronym I’m familiar with.

            • Passerby6497@lemmy.worldEnglish
              2 hours

              Incident response, basically getting pulled in to clean up someone else’s fuck up that got them owned.

    • A_Random_Idiot@lemmy.worldEnglish
      1 day

      why is any critical control equipment internet accessible.

      I understand some machines might need internet to send reports and make document filings and shit. But it should not be the critical control equipment that makes the system function. And there should be a massive physical gap between the internet machines and the critical machines.

      Jesus christ, even by the abysmally low bar I have for humanity, it still finds ways to dig itself a path under it.

      • Uriel238 [all pronouns]@lemmy.blahaj.zoneEnglish
        17 hours

        The entire tech sector screamed about the vulnerability of IoT devices, plenty of which are now participants in zombie botnets.

        Does anyone remember in 2015 when YESCO billboards by the thousands were hacked to show Goatse instead of an AT&T ad? Pepperidge Farm remembers. IoTs are still vulnerable, including big municipal ones and, yes, Flock Safety cameras.

        And CISA was severely broken had its staff severely reduced by DOGE.

        In unrelated news the CDC, also a victim of DOGE, is entirely unprepared for the next epidemic.

    • BoofStroke@sh.itjust.worksEnglish
      22 hours

      Why in the everloving fuck would scada ever be connected directly to even the regular lan, let alone the Internet!?!

      • NoWay@lemmy.worldEnglish
        21 hours

        Laziness. Poor training. Stupidity.

        Why wouldn’t you want a key utility controlled from around the global?

    • meathorse@lemmy.worldEnglish
      1 day

      A lot of this is often down to shitty IT practices by the vendor who has little IT knowledge let alone decent security insight.

      I helped support a new PCL installation (technician control PCs and software upgrade) in this exact scenario many years ago. The vendor came in expecting to treat our system as their own. Direct remote access to several over-specced, physical servers from the outside world, available on our core network, shared account names (and passwords!) across all devices, every account a local admin and their own domain admin account so they can make changes as required… Needless to say our security guy almost broke a rib from laughing so hard!

      What they got was very different. I ended up doing fairly basic troubleshooting to get the shitty software working without it needing admin rights to run - XP era software running on Win7 clients so it was meeting basic folder and reg key write permissions - everything else was locked down as tight as we could. They didn’t even get admin on the servers.

      A few years later I heard they’d reported that our site was their most reliable. Probably due to the fact that they (or the techs) couldn’t fuck with anything without our permission. Apparently their original configuration is normal in the industry… Shudder

    • skvlp@lemmy.wtfEnglish
      1 day

      Yup, Sleepy-Don really do have the whole Iran situation under control.

    • Uriel238 [all pronouns]@lemmy.blahaj.zoneEnglish
      18 hours

      According to our peerless leader, I blame it on Minnesota because they are grossly incompetent

      But then he tells California (and Canada) we are to blame for wildfires for our failure to rake our forests.

    • khepri@lemmy.worldEnglish
      24 hours

      So are we just kind of admitting that there exists no way to expose any networked device to the internet securely? Because if it’s not possible for PLCs I don’t see why it would be possible for any device. If water utilities have to take these offline, then how does that advice not apply for every internet-capable device in every commercial and industrial facility worldwide?

      • emergencyfood@sh.itjust.worksEnglish
        16 hours

        There are ways, but they are costly and complicated and do not stop all attacks. So the golden rule is that if critical infrastructure doesn’t need to connect to the internet, it shouldn’t.

      • Archr@lemmy.worldEnglish
        23 hours

        I don’t work directly with PLCs but we do have them at my work. The main thing that makes these different from any other devices is that they typically control physical machines. Which means there is a real danger that them becoming compromised could lead to damaged equipment or even death.

        Additionally, many PLCs use older OS versions like xp because their software is notoriously out of date (if it’s working why rewrite it when that could introduce safety concerns)

        This “urgent” message from CISA does not prove that any of these devices are internet accessible just that if they are then they should be removed. If CISA wanted to require this then they would have released a binding operational directive (BOD) or an emergency directive (ED). Both of which are publicly viewable on their site.

      • historicaldocuments@lemmy.worldEnglish
        16 hours

        So are we just kind of admitting that there exists no way to expose any networked device to the internet securely?

        It’s done all the time. It’s not a product you can buy, though, it’s a process. There’s probably a good “go by” for how to start the process for citywide infrastructure, but it’s just one more document for people to ignore.

    • Zwuzelmaus@feddit.orgEnglish
      1 day

      disconnect […] from the internet

      And these guys really think that this is doable as easy as snapping your fingers?

      hmm…

      And these guys really are national advisors on things related to computers and internet??

      hmmmmm…

      • Godort@lemmy.caEnglish
        1 day

        The specific advice they give is to disconnect all PLCs and only access them through a VPN, enable password protection and change the default passwords, and only allow IPs from known engineering laptops and other critical assets.

        This is absolutely doable and any IT professional could manage this change. The only barrier that prevented this from happening before is that you need buy-in from someone with access to the purse strings