300+ Water Treatment Plant SCADA systems Exposed Online

Executive Summary

Properly treated wastewater is critical for disease prevention, environmental protection, and safe drinking water to protect public health and all industrial activity. This firmly places Water and Wastewater facilities under the umbrella of Critical Infrastructure.

The ability to supply water and manage wastewater are essential functions jointly performed by the government as well as the private sector. Their disruption, corruption, or dysfunction would have a crippling effect on national security, the national economy, national public health and safety – or a combination of all of these factors.

As these facilities are crucial for every state and country, Advanced Persistent Threat (APT) groups and malicious hackers target these facilities with various motives, which we have observed in prior attacks in this sector as well.

Notable recent cyber events related to the water infrastructure are:

1. Tampering of the Post Rock plant, Kansas

WYATT A. TRAVNICHEK, 22, of Ellsworth County, Kansas, was charged with tampering with a public water system and recklessly damaging a protected computer during his illegal access.

Travnichek wilfully gained unauthorized access to the Ellsworth County Rural Water District’s secured system. Through the course of this unlawful access, Travnichek allegedly committed acts that shut down the facility’s processes, affecting the facility’s cleaning and disinfection procedures, causing injury to the Ellsworth Rural Water District No. 1.

Travnichek was hired at the Post Rock plant in January 2018. He was given the authority to monitor the facility via a remote computer login system. Despite no longer working at the plant, he was able to access the network on March 27, 2019, to disrupt the water treatment process.

“By illegally tampering with a public drinking water system, the defendant threatened the safety and health of an entire community,” said Lance Ehrig, Special Agent in Charge of EPA’s (Environmental Protection Agency) Criminal Investigation Division in Kansas.

• San Francisco Bay Water Treatment Plant

On January 15, 2021, a Threat Actor used the login credentials of a former employee at the facility to launch an attack on an unidentified water treatment plant in the San Francisco Bay Area.

The hacker could carry out this attack since he gained access to a former employee’s TeamViewer account, which allows employees to use their laptops remotely within the water treatment plant site.

• Oldsmar, Florida Water Treatment Facility

An anonymous hacker gained access to the computer systems of the water treatment facility in Oldsmar, Florida, and altered chemical balance to unsafe levels.

On Friday, February 5, 2021, the attacker gained access to a computer system that was set up to allow remote control of water treatment processes and tried to change the lye concentration.

Oldsmar Sheriff Bob Gualtieri, in the press statement, mentioned that the hacker tried to manipulate the sodium hydroxide concentration in the water.

“Sodium hydroxide, also known as lye, is the main ingredient in liquid drain cleaners. It’s also used to control water acidity and remove metals from drinking water in the water treatment plant.“

“The hacker changed the sodium hydroxide from about 100 parts per million to 11,100 parts per million. This is a significant and potentially dangerous increase.”

• Ransomware attacks on Water Treatment Facility

The Joint advisory released on October 14, 2021, by the Federal Bureau of Investigation (FBI), the Cybersecurity and Infrastructure Agency (CISA), the Environmental Protection Agency (EPA), and the National Security Agency (NSA) highlighted several malicious cyber activities by both known and unknown threat actors targeting the Information Technology (IT) and Operational Technology (OT) networks, systems, and devices of U.S. Water and Wastewater Systems (WWS) Sector facilities.

• August 2021. Malicious cyber actors deployed Ghost ransomware against a California-based WWS facility. The ransomware variant had been in the system for about a month and was discovered when three Supervisory Control and Data Acquisition (SCADA) servers displayed a ransomware message.

• July 2021. Threat actors used remote access to introduce ZuCaNo ransomware onto a Maine-based WWS facility’s wastewater SCADA computer. The treatment system was run manually until the SCADA computer was restored using local control and more frequent operator rounds.

• March 2021. Threat actors used an unknown ransomware variant against a Nevada-based WWS facility. The ransomware affected the victim’s SCADA system and backup systems.

The cyber incidents mentioned above indicate that malicious hackers are well versed in the Operational Technology (OT) functioning within the water sector and are aware of the damage they can cause by launching cyberattacks on WWS.

On January 27, 2022, the Biden Administration announced the Industrial Control Systems Cybersecurity Initiative – Water and Wastewater Sector Action Plan to strengthen national cybersecurity defenses and information-sharing.

According to a White House statement,  the Industrial Control Systems Cybersecurity Initiative Water and Wastewater Sector Action (Water Sector Action) Plan is a collaborative effort between the federal government and the critical infrastructure community to facilitate the deployment of technologies and systems that provide cyber-related threat visibility, indicators, detections, and warnings.

This initiative shows that government organizations have been actively preparing for cyber-attacks against critical infrastructure. Advanced Persistent Threat (APT) groups, malicious hackers, and suicide hackers have been targeting the flaws found in the components mentioned below for a long time for their varied motives and cyber-attacks on Water and Wastewater Facility (WWF).

There are many components within WWF. A few of them are mentioned below:-

• Physical Elements – Valves, distillation tankers, reservoirs, pipes, pumps, motors, PLC, etc
• Cyber Elements – Supervisory Control and Data Acquisition.
• The Human Element – Employees, contractors, managers, etc.

These components work collectively to provide the desired output within the WWF. During Cyble Research Lab’s routine threat monitoring, we came across 308 Water & Wastewater treatment facilities’ SCADA systems that are exposed over the internet. At the time of publishing this report, these systems are still exposed over the internet.

The SCADA systems play an important role in monitoring real-time, processing data, directly interacting with sensors, valves, pumps, motors, etc., and controlling the WWF operations.

Using SCADA systems, the plant operators can remotely access the site utilities and ensure smooth operations throughout the facility. As these systems are the heart of critical infrastructure, malicious hackers can attack these systems to compromise the entire WWF facility.

A malicious hacker gaining access to SCADA systems can bypass the authorization by launching a brute-force attack, using unchanged factory default credentials, credential stuffing, SQL injection, DDOS, etc.

Findings

A malicious hacker bypassing the authorization of the SCADA system can gain complete control over the system, allowing them to change the Aeration Basin settings & Detention Tank settings.

Aeration is the most important component of an activated sludge treatment system. The volume of wastewater treated by a well-designed aeration system is directly proportional to its design. 

The detention basin is used in the wastewater treatment plant to smooth out oscillations in the wastewater inflow, ensuring an equal flow throughout the facility’s process tanks. 

An attacker with administrative access to the SCADA system can change the aeration and detention basin parameters. These parameters are critical to keeping the treatment operations running optimally and meeting discharge criteria.

Wastewater goes through various stages, including treatment in Chemical Feed, designed for automated chemical injection into the chemical feed tank for water or wastewater treatment.

A hacker gaining access to the SCADA system can manipulate or turn off the alarms set by the operator for the discharge of chemicals such as phosphorus and phosphate (as shown in Figure 3). This can cause a steep rise or decline in the pH levels of the water being treated.

Gaining access to the SCADA system can provide a malicious hacker with complete visibility of the process of the water treatment facility and any connected pump houses.

This might help hackers target a specific component within the water treatment channel and understand the complete water treatment flow for a particular state, as shown below.

Attackers with access to the SCADA system can change the setpoints set by the plant operator, which might result in the malfunctioning of equipment currently being used for controlling water pressure and its flow, as shown below.

A hacker can abruptly START/STOP the valves and pumps. This may result in a “water hammer,” causing damage to the pipelines connecting to different essential components, as shown in Figures 6 & 7.

Water storage tanks are generally connected to a pump used for pumping water. An attacker gaining access to the water treatment pump (as shown in Figures 8 and 9) using this process through the SCADA system can crush the water tank due to external pressure by changing the flow state of the pump.

Attackers can also manipulate the standard pressure and flow values which might hinder the smooth operation of water distribution for a state, as shown in Figure 10. As all the SCADA systems are designed differently depending on the need and operations of the plant, the impact of manipulating these settings may vary.

The attacker can retrieve sensitive data such as user ID, passwords, WTP reports, etc., through a compromised SCADA system, as shown in Figures 11 & 12. Since the SCADA system also provides the feature to add users, attackers can create a user account to gain privilege. Also, they may enable or disable the current user privileges, locking authorized personnel out of the compromised SCADA system.

Impact

• Water and Wastewater Treatment Facilities use various water treatment chemicals like pH neutralizers, anti-foaming agents, coagulants, and flocculants. An attacker gaining access to the Water Treatment Plant SCADA systems can manipulate the chemical levels set by the plant operator, which might make the water being distributed unsuitable for household, industrial or agricultural usage.

• An attacker gaining access to the SCADA system of Water treatment plants can Start or Stop the pumps and valves operating within the facility, which may cause a “Water Hammer,” a subset of transient flow or surge analysis.

It is a special scenario where there is a quick shift in inflow velocity. Typically, this happens when a pump or valve closes abruptly. A Water Hammer may cause very high-pressure transients, which can break pipes and cause pipeline vibrations.

• An attacker’s manipulation of control operations within the treatment plant can result in the collapse of the water tank as they are designed to withstand a small amount of internal pressure and no external vacuum pressure.

• The water and wastewater treatment plant environment deals with equipment working in extreme environments like rapid water flow, pump operations, etc. Any interruption in the water treatment plant process can risk the safety of employees or operators working in the facility.

• Upon gaining access to the SCADA system of a water treatment plant, an attacker can change the parameters that the plant operator sets to raise the alarm to the main operator if the water levels of the reservoir or tank decrease or increase past a certain level. If the plant operator is not aware of these settings or ignores the changes made to these settings by a hacker, this can set in motion a chain of disastrous events.

• Water and Wastewater Treatment facilities are crucial for countless households, businesses, industrial operations, and processes happening in a particular state. A cyber-attack on the facility can temporally stop these operations.

• SCADA systems within water treatment plants provide a lot of confidential and sensitive data about the equipment and the users of a system such as usernames, passwords, plant processes, equipment, etc. There is a high possibility that a Threat Actor (TA) can sell this intelligence on the Dark Web or Deep Web forums.

• The SCADA systems observed during the investigation also provide the option to create a new user. An attacker can abuse this feature and create an unknown user account gaining persistence into the SCADA system.

• SCADA systems within the water treatment plant can provide information regarding the ICS networks that they are placed in, which might provide attackers with sufficient data to plan and execute further attacks upon the organization.

• In the outbreak of war or during a terror attack, adversary state hackers can target these systems to create chaos among the civilian population and state bodies.

• Water and wastewater treatment provide access to home water, and sanitation directly influences household finances and, eventually, the national economy. For example, having safe drinking water and a toilet at home may help a family save money on health care. Cyberattacks on this critical infrastructure can impact the country’s overall economy.

• Organizations providing water and wastewater management can suffer a tremendous monetary and reputational loss due to cyber-attacks.

• Critical services such as fire protection, healthcare, and heating and cooling processes would also be disrupted by the interruption or cessation of drinking water service, resulting in significant consequences to national or regional economies.

Conclusion

Water and Wastewater treatment facilities play a key role in national health, safety, and the economy. Hackers gaining access to systems used to monitor, control, and manage plant operations can launch cyber-attacks that can have disastrous consequences for the entire nation.

Exposing SCADA systems over the internet with no or minimum authorization can allow hackers to manipulate operations that should solely be operated by the facility engineer(s). This can also cause loss of life, physical damage to equipment, monetary loss, reputational loss, water scarcity in a particular region, etc.

Cyble Research Labs believes that organizations responsible for the water treatment, supply, or distribution must immediately evaluate their IT and OT equipment vulnerabilities and their exposure.

Recommendations

We have listed some of the essential cybersecurity best practices that create the first line of control against attackers. We recommend that our readers follow the best practices given below:

1. Implement strong network segmentation between IT and OT networks to prevent malicious attackers from pivoting to the OT network after compromising the IT network.
2. Do not expose critical assets from the internet. If the assets need to be connected to the internet, ensure the appropriate authorization exists to prevent attackers from gaining access to these systems.
3. Ensure that all remote access technologies have logging enabled and regularly audit these logs to identify instances of unauthorized access.
4. Install independent cyber-physical safety systems. 
5. Update software, including operating systems, applications, and firmware on IT network assets.
6. Implement regular data backup procedures on both the IT and OT networks.
7. Enforce a strong password policy.
8. Ensure that the organization’s emergency response strategy considers the whole spectrum of potential implications that cyberattacks can have on operations, such as loss or manipulation of view and control and safety risks.
9. Personnel in charge of monitoring WWF should look out for the following unusual actions and signs, which may indicate threat actor activity:
10. Unexplained SCADA system restarts.
11. Dynamic parameter values that do not change
12. Access to SCADA systems at unusual times
13. Access to SCADA systems by unauthorized/unrecognized individuals
14. Operators and engineers should be trained to spot and report phishing and social engineering efforts through awareness and simulations.

Disclaimer  

All information provided in this blog is for general information and educational purposes only and for no other purpose. It is not intended and should not be construed to constitute advice of any nature whatsoever. The information contained herein may not be applicable to all situations and may not reflect the most current situation. Nothing contained herein should be relied on or acted upon without the benefit of independent advice based on the particular facts and circumstances presented, and nothing herein should be construed otherwise. The contents, opinions, and findings rendered are subjective. Cyble reserves the right to modify the contents of this blog at any time without prior notice. Although reasonable efforts have been made to include accurate and up-to-date information herein, however, Cyble makes no warranties or representations of any kind as to the accuracy, correctness, currency, or completeness of all the contents stated in the blog.  

The contents of the blog, any discrepancies or differences of any nature whatsoever, are not binding and have no legal effect for compliance or enforcement purposes or for any other purpose. You agree that access to and use of, and reliance in any manner on this blog, including all the contents thereof, is at your own risk. Cyble disclaims all warranties of any kind, express or implied. Neither Cyble nor any party involved in researching, creating, producing, or delivering this blog or anything related thereto shall be liable for any consequence, loss, or damage, including direct, indirect, special, consequential, loss of business profits, or special damages, whatsoever arising out of access to, use of, or inability to use, or in connection with the use of this blog in any manner, or any errors or omissions in the content thereof.” 

Exposures found during the research have been reported to respective CERTs.