Read the original article: Shining a Light on SolarCity: Practical Exploitation of the X2e IoT
Device (Part One)
In 2019, Mandiant’s Red Team discovered a series of vulnerabilities
present within Digi International’s ConnectPort
X2e device, which allows for remote code execution as a privileged
user. Specifically, Mandiant’s research focused on SolarCity’s (now
owned by Tesla) rebranded ConnectPort X2e device, which is used in
residential solar installations. Mandiant performs this type of work
both for research purposes and in a professional capacity for their
global clients.
Mandiant collaborated with Digi International and SolarCity/Tesla to
responsibly disclose the results of the research, resulting in the
following two CVEs:
-
Hardcoded
Credentials (CVE-2020-9306, CVSS3.0: 8.8) -
Execution
with Unnecessary Privileges (CVE-2020-12878, CVSS3.0:
8.4)
Technical details can be found in Digi
International’s 3.2.30.6 software release, and on FireEye’s
Vulnerability Disclosures GitHub project (FEYE-2020-0019
and FEYE-2020-0020).
This two-part blog series will discuss our analysis at a high level,
explore the novel techniques used to gain initial access to the
ConnectPort X2e device, and share the technical details of the
vulnerabilities discovered. Topics to be covered will include physical
device inspection, debugging interface probing, chip-off techniques,
firmware analysis, glitch attacks, and software exploitation.
If you’re interested in continuing the story in Part
Two, you can read it now.
FAQ
What devices are affected, and (potentially) how many devices are affected?
The vulnerabilities described in this post affect ConnectPort X2e
devices as well as the SolarCity rebranded variant. Other vendor
devices may also be vulnerable. It is unclear how many ConnectPort X2e
devices are deployed in the wild.
How is the issue being addressed?
Mandiant worked independently with Digi International and Tesla to
remediate the vulnerabilities. Mandiant would like to thank Digi
International and Tesla for their cooperation and dedication to
improving the security of their products.
How would an attacker exploit these vulnerabilities?
An attacker with local network access (such as being connected to an
individual’s home network via Ethernet) to a vulnerable X2e device can
exploit CVE-2020-9306 and CVE-2020-12878 to gain privileged access to
the device.
Who discovered these vulnerabilities?
Jake Valletta (@jake_valletta), Sam Sabetan (@samsabetan)
More information such as videos and datasheets on Mandiant’s
Embedded Device Assessments can be found here.
Technical Analysis
Device Overview
Before diving into the details, we’ll discuss the ConnectPort X2e
device (referred to as X2e device throughout the post) at a high
level. The X2e device is a programmable gateway that connects to and
collects data from ZigBee devices. It is commonly used as a Smart
Energy gateway to interpret and send energy readings from a
residential Solar Inverter. Vendors will often purchase an X2e device
and configure it to read power consumption generated by a customer’s
Solar Inverter. Figure 1 outlines a typical residential solar
installation and highlights the X2e’s role.
Figure 1: Typical X2e residential deployment
For our research, we focused on the X2e device used by SolarCity,
now Tesla, to retrieve data from residential solar installations. A
typical setup would involve SolarCity providing a customer with a
gateway that would be connected to the Internet via an Ethernet cable
on the customer’s home network. Figure 2 shows one of the SolarCity
branded X2e devices that we tested.
Figure 2: X2e device
Without even plugging in the X2e device, we know of at least two
separate interfaces to explore: the Ethernet interface and the ZigBee
radio. Note that we did not review the ZigBee interface between the
X2e and a solar invertor, and that interface will not be covered in
either Part One or Part Two of this series.
Initial Analysis and Physical Inspection
Network Reconnaissance
We started our research by assessing the X2e device from a network
perspective. By using nmap, we discovered that the device exposed
both SSH and HTTP/HTTPS, shown in Figure 3.
Figure 3: Port scan results from the X2e
Upon accessing these services remotely, we noted that both services
required authentication. We also performed limited brute force
attempts, which were unsuccessful. Additionally, the underlying
services were not vulnerable to any public exploits. With not many
network-based leads to follow, we shifted our analysis to a hardware
perspective to determine if any local attacks may be possible to gain
initial access onto the device.
Physical Board Inspection
To begin our hardware analysis, we removed the plastic casing from
the device and mapped out the various integrated circuit (IC)
components and searched for potential debugging interfaces.
Inventorying the components present on the circuit board (also known
as a PCB) is a crucial step in understanding how the device was
designed and what can be expected down the road. Figure 4 shows the
mapped-out components as well as a cluster of pins that resembled a
typical 3-pin universal asynchronous transmit/receive (UART)
connection, a common debugging interface on embedded devices.
Figure 4: X2e components and suspicious
cluster of pins
Without a remote connection to the X2e device, UART is an attractive
target. UART typically provides the equivalent functionality of a
service like SSH or Telnet and the added benefit of watching verbose
output during system boot. To determine if the cluster of pins was a
UART interface, we first soldered a 3-pin through-hole header to the
PCB. Using a combination of continuity tests with a multimeter and the
digital logic analyzer Saleae, it became
apparent that we were in fact dealing with a UART interface. The
Figure 5 shows the three pins (Ground, TX, RX) connected to the
header. Attached to the other end of the three wires was a FTDI
serial TTL-232 to USB adapter, which was connected to a Linux
virtual machine.
Figure 5: Connecting to potential UART interface
In addition to correctly identifying the UART pins and a UART to USB
adapter, we also needed software to read/write from the interface as
well as knowledge of the baud rate. Baud rates vary but typically
follow standard values, including 9600, 14400, 19200, 38400, 57600,
and 115200. Using the python module pySerial, we connected to the USB adapter and
tried standard baud rates until one of the rates produced readable
ASCII output (an incorrect baud rate will typically produce
non-readable output), and determined the X2e used a baud rate of 115200.
Upon booting the X2e, we noted output from the BootROM, bootloader
(which was Das U-Boot 2009.8, a common embedded bootloader), as well
as output from the Linux kernel transmitted over the UART connection,
shown in Figure 6.
Figure 6: UART boot messages
Many configurations of U-Boot allow a physically connected user
(using an interface such as UART) the ability to interrupt the boot
process; however, this configuration explicitly disabled that feature,
shown in Figure 7.
Figure 7: Uninterruptable U-Boot
bootloader on the X2e
Interrupting a bootloader is attractive to an attacker, as often the
boot parameters passed to the Linux operating system can be
manipulated to control how it will load, such as booting into single
user mode (typically a recover shell) or mounting filesystems as
read-write. In the case of the X2e, the UART connection was mapped to
a Linux TTY which required username and password authentication, shown
in Figure 8.
Figure 8: User authentication to Linux
over UART
Without any ability to interrupt the boot process or credentials to
authenticate to the X2e, we were faced with another dead end. We then
shifted our analysis to obtaining the firmware stored on the X2e’s
non-volatile storage.
Chip Removal and Data Extraction
In this section, we’ll cover the basics of non-volatile memory,
often referred to as “flash memory”, present on embedded devices as
well as the process used to extract content from the chip. As
mentioned, taking inventory of the components on the PCB is an
important first step. Figure 9 shows the suspected flash chip present
on the PCB magnified under a digital microscope.
Figure 9: Closeup of Spansion flash
The visible markings seen in Figure 9 are important as they allow us
to determine the manufacturer and model of the flash, which will
assist us with obtaining the datasheet for the chip. In our case, the
NAND we were dealing with was a Spansion S34ML01G1, and its datasheet
could be found here.
NAND Overview
Before we talk about acquiring the firmware from the NAND chip, it’s
important to first understand the various scenarios that embedded
devices typically follow.
NAND verses NOR: These fundamentally different technologies
each have their own benefits and drawbacks. NAND is cheap but suffers
from high probability of “bad blocks,” or areas that are corrupt
sometimes directly from the factory. As such, protections and
considerations need to be present to be able to protect against this.
NAND is also much faster to erase and write, making it ideal for
storing file systems, kernels, and other pieces of code that may need
to be reset or changed. NOR has significantly faster read times but is
not as flexible with accessing data and has low erase and write
speeds. NOR is usually used for low-level bootloaders, hardcoded
firmware blobs, and other areas that are not expected to change
frequently. The X2e uses a NAND flash.
Serial verses Parallel: This refers to how the data is accessed
and is typically visually identifiable. If there are a large number of
pins, the flash is likely parallel. Serial NOR chips can be small in
size and typically need eight or fewer pins to function. Common serial
interfaces are Serial Peripheral Interface (SPI) or Inter-Integrated
Circuit (I2C), while a common parallel interface for NAND is Open NAND
Flash Interface (ONFI2.0, ONFI3.0). The X2e is a parallel flash.
IC Form Factor: Another visually identifiable trait—form factor
(or “package”)—refers to how the chip is attached to the PCB. There is
a long list of options here,
but common surface-mount flash packages include small outline package
(SOP), thin outline small package (TOSP), or a variant of ball grid
array (*BGA). The key distinction here is SOP and TOSP expose the
pins, while BGA conceals the pins under the package. The X2e is
BGA63, also referred to as a 63-pin BGA package.
Managed verses Unmanaged Flash: This one is more applicable to
NAND, for reasons alluded to in the NAND verses NOR section. As
stated, NAND needs help to manage the integrity of the data. With
unmanaged NAND, the IC reserves sections of the flash (often called
“spare” area) for someone else to manage the data. This is
typically implemented as either a kernel driver or an external NAND
controller. Managed NAND means that the IC package includes the
controller and transparently manages the data. This is extremely
common in embedded devices, as either embedded MMC (eMMC) or universal
flash storage (UFS). The X2e uses unmanaged flash and is
controlled by the main microcontroller present on the PCB.
With the basics out of the way, we proceeded with physically
removing the chip from the PCB.
Chip Removal
Physical chip removal is considered a destructive approach but can
certainly be performed without damaging the PCB or the flash chip
itself. When presented with removal of BGA packages, the two most
common removal techniques are either hot air or infrared light (IR).
Commercial solutions exist for both hot air and IR, but cheaper
options exist with hot air removal. We opted to use hot air on the X2e.
To minimize damage to the PCB and flash, a PCB heater or oven can be
used to slowly bring the entire PCB to a temperature right below the
solder melting point. This will reduce the amount of time we need to
focus our hot air directly onto the flash IC and help with reducing
the heat dissipation into the PCB throughout the process.
One final trick that can be used to minimize nearby chips from being
damaged or lost (due to the air pressure) is the use of high-heat
resistant tape, commonly referred to as Kapton tape. Figure 10 shows
the PCB wrapped in Kapton tape to protect nearby components.
Figure 10: High-heat resistant tape on PCB
Figure 11 shows an example setup with the X2e PCB inserted into a
PCB heater, with a hot air gun suspended over the IC.
Figure 11: Hot air rework/reflow s
[…]
Read the original article: Shining a Light on SolarCity: Practical Exploitation of the X2e IoT
Device (Part One)