This week I received an Amazon Prime Day purchase EVGA NVIDIA GeForce RTX 3090, but there was a catch. The GPU being sold as “new” by Amazon.com seemed to have been a customer return that had its security seals broken and had another card substituted in its place. While Amazon has been good about the return thus far, this experience has profound hardware security and hardware supply chain security implications.

For this, we also have a video version that you can find here:

There are a few photos that are in that version that are not in this, just given the timing of both. As always, we suggest opening the video in its own tab, browser, or app for a better viewing experience.

Followers of my personal account on Twitter this week may have seen the tweet that came out in disbelief of what happened. On Amazon Prime Day I ordered a new EVGA NVIDIA GeForce RTX 3090 FTW3 Ultra edition and things did not go as planned.

Scammed by Amazon. Bought a new @TEAMEVGA @NVIDIA RTX 3090 (Amazon seller.) It arrived. Box seals broken. Inside was a RTX 3070. 🙁 pic.twitter.com/ECMyn0eEfP

— Patrick J Kennedy (@Patrick1Kennedy) July 21, 2022

On July 12, 2022, I ordered the GPU. Although NVIDIA sent us our original GeForce RTX 3090 for review, we needed another one for a test platform for several of our other series on STH. The EVGA card was sold by Amazon.com Services LLC and was listed in New condition. This was purchased via the Add to Cart/ Buy Now box, not by looking for used units.

The unit shipped on July 13, 2022, and then things started to go awry.

First, the unit was shipped, requiring a signature delivery. I was heading to Zurich, Switzerland that weekend, but according to the estimated arrival date, I was to be around for the shipment. Instead, Amazon batched the card with other items so it instead would arrive after I left for Europe. That meant I had to spend $10 to get UPS to deliver the package at a later date when I could be there to sign for it. When the package arrived, it looked like it had a tough life.

It seems as though this package was open at some point during transit. When it was opened, it appeared that someone had folded the flaps over each other. Then a copious amount of clear plastic tape was used to seal the box. You can see it on the doorstep at delivery in the video, but the opposite side was opened to gain access to products just to preserve this packaging masterpiece.

The contents seem to have arrived, but the GeForce RTX 3090, destined for a test node, had a curious feature. EVGA uses security tape to seal its packaging. On this package, both EVGA seals were already broken.

In addition to the two broken seals, there was a LPN PM sticker. Often Amazon uses the LPN RR stickers for returns put back into stock, so this looked similar. Also, there is a sticker under that LPN sticker with the model information and a “NewItem” printed on the label.

Here is another look at the label on the end of the box, this time opened.

Inside the box was a normal inside box with the foam carrier.

Removing the top foam made us see the ICX3 cooler style that EVGA uses on a number of its cards. If someone processing a return saw this, they would see the card on the front of the box and assume the same card was sitting there.

Even the EVGA badge and the non-model specific EVGA GeForce installation guide was included.

EVGA uses branded ESD bags, and this card was using the EVGA bag with an ESD seal still on the bag.

The one view, other than tracing the serial numbers that would give this away was looking at the top of the card that says “RTX 3070” instead of “RTX 3090”. That single 7 versus 9 digit change is the only giveaway one would reasonably see when inspecting this package to tell that this was the wrong card.

That “3070” is often hidden by the foam carrier in the box. Being empathetic, it is not too hard to understand how this could happen. Someone that did not know the difference between a RTX 3070 and a RTX 3090 would miss this if they were under time pressure to validate a previous return. When opening the box and seeing the card that looked like the box cover, I originally assumed it was the right card, and I saw the ESD sticker sealing the card. It was only later that I realized this was a 3070.

My best guess is that someone purchased an EVGA RTX 3070 and a 3090. They then kept the 3090 and returned the 3090 packaging with the RTX 3070 version inside. Amazon processed the return of an expensive card and the person checking missed the 7 versus 9 on the top of the card, and did not have a way to validate serial numbers. They then saw the ESD bag seal and decided that it could go in new stock instead of a used queue.

I opened the return with Amazon and spoke to someone on the phone to explain what happened and this card is now on its way back to Amazon. I wanted to get on the phone because I was sending back a RTX 3070 in a 3090 box, and if someone does proper checks, they will see this is not the card that matches the packaging. I was a bit apprehensive about doing this type of return since it is a mismatch.

In the end, assuming the refund is processed, I will be out about a week putting together the system we will use to test the Project TinyMiniMicro nodes and for our STH Mini PC series. I also have the risk of having to do the return and having an issue because there is a mismatched product. Further, I had to stop by the UPS store and drop off the package, and that is a 15+ minute one-way trip. The positive is that pricing on the RTX 3090 has fallen since then, so I probably will end up a bit better off with the return other than all of the lost time.

The eye-opening impact is really the hardware security aspect of this story.

Bloomberg’s discredited spy chip article and a follow-up piece that I interviewed their primary source who disagreed with Bloomberg’s reporting, while fake news did something for the industry. It helped highlight the challenges with hardware security and supply chains. This story is a more practical and dangerous story from a hardware security perspective.

One of the big challenges with hardware security is that items can be intercepted during shipping. We assume this was just a package that Amazon did not properly fill and that got crushed and re-taped by UPS. Compared to some stories of systems being intercepted during shipment, this is a poor cover-up job. Still, it is not completely uncommon to see Amazon packages take a few bumps along the way.

The bigger concern is that it seems like a unit sold by Amazon.com and not a 3rd party seller was not new. Instead, even with security seals being broken, it was placed back into NewItem stock and sold as “New” per the invoice above.

Taking a step back for a moment, if the card was actually an EVGA GeForce RTX 3090 and with the ESD bag still sealed, I probably would have used it being in a time crunch to get the new system online. If the entire card can be substituted and go through Amazon’s return logistics process, ending up as “new” stock, then what if someone had more nefarious intentions? What if someone tampered with the GeForce RTX 3090 and got it accepted back as new stock by Amazon?

The downstream buyer might use the compromised RTX 3090 in their system. This is a card where the PCB is not exposed since there are fans, heatsinks, and shrouding covering the entire card. It would be impossible without disassembly to see if the card was tampered with before installation. That compromised RTX 3090 would be plugged into the PCIe slot in a motherboard. PCIe devices physically installed into motherboards have very low-level system access due to their natures. This is perhaps one of the scariest types of attacks since it would allow a huge amount of access to a system.

The key here is that by taking a previously sold computer product and reselling it, by Amazon.com as the seller as “new,” most buyers have the reasonable expectation that the unit went from the manufacturer to a distributor to Amazon or directly to Amazon. Having the potential of muddling returned computer products into this new item pool means that that chain of custody cannot be guaranteed.

Instead, we had a third party tamper with the GPU Amazon.com Services LLC sold us as new. Then, something happened in shipping leading to the package being accessible en route. This story, given Amazon’s large presence, should make computer hardware security experts very nervous as it has all of the elements to be disastrous for end users.

Again, the chain of events is simple to understand. I am a bit disheartened by the inconvenience of this. I will say that Amazon has been easy to initiate the return on so I have no complaints at this point on the return process (we may have an update if that changes.)

Still, given Amazon’s market presence and the fact that it is letting highly tampered products into its supply chain and selling them as new, the hardware security aspects of this are mind-boggling. For Amazon, the challenge is that the alternative is that taking returns on these products and automatically not putting them into a used/ refurbished stock can be costly. However, if this practice continues, it becomes hard to trust one of the world’s largest computer component resellers. I doubt AWS would accept this in its hardware supply chain, so there is a bit of a “do unto others” that I cannot escape feeling here.

Originally, I was not going to write up this story, much less do a video. The Twitter post was all I planned on doing. At the same time, I think that our readers need to be aware that something like this can happen. From a hardware security and supply chain perspective, the box was bad, but there is little one can do to show tampered hardware can get through into a reseller’s “new” stock more than having a product that has been substituted in the package.

For the industry and our readers, Amazon’s computer component supply chain being this compromised and passing tampered packages as new and sold by Amazon, not a third party seller, should be a wake-up call. This is as bad, or worse, than what we found in Dude this should NOT be in a Dell Switch or HPE Supercomputer.

Save my name, email, and website in this browser for the next time I comment.

Sign me up for the STH newsletter!

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Get the best of STH delivered weekly to your inbox. We are going to curate a selection of the best posts from STH each week and deliver them directly to you.

By opting-in you agree to have us send you our newsletter. We are using a third party service to manage subscriptions so you can unsubscribe at any time.