LLMs industrialize the con, and single sign-on becomes the master switch
When faking a job interview, monitoring an inbox and synthesizing a voice all cost close to nothing, anyone's SSO account can be taken over silently; who the bank makes absorb the loss matters more than teaching users to spot phishing.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
You have not been hit yet only because the con is still too expensive
Manish's starting point is that LLMs are attacking the cost side of running a complicated scam. The old Nigerian-prince emails let people conclude ‘I'm smart enough, I'd never fall for that’. But today the drudgery — building a fake company website, staging a whole interview process, reading through a victim's email one message at a time — can all be automated and done cheaply. Note that this is not the future; it is already happening.
— ManishOne SSO account is enough to pry open every asset you own
Because users will always reuse passwords, the industry shifted toward identity providers like Google and Apple, and SSO became ‘one ring to rule them all’. Break into that identity account and you have the email inbox at the same time; with the inbox you can reset the passwords on bank and brokerage accounts. So attackers are in no hurry to cash out. They sit quietly on the account, watch your financial rhythm over a long stretch, and move when there is a lot of money in it.
— PatrickImpersonating a type of person costs pennies; impersonating you is what's expensive
Hong Kong has already seen a 25 million US dollar deepfake fraud: an employee saw the ‘CEO’ and other executives on a video call and was told to push the buttons that sent the money out. Cloning one specific person still requires more than thirty minutes of that person's voice. But if all you need is a generic profile — ‘a man in his forties with a New England accent’ — the cost can come down to a few cents. And remember: today's technology is the worst it will ever be.
— ManishThe tighter banks seal the pipes, the more money runs through gift cards
Banks are required to block illicit money flows, so scammers migrate to payment instruments at the regulatory margins. A single gift card can move 500 US dollars anywhere, and repeating that same move 100 times moves 50,000 US dollars — and gift card companies are exactly the firms that are bad at detecting a run of consecutive operations. Bitcoin ATMs carry absurd premiums, so no normal user would touch them; the only reason to be standing at one is that a scammer told the victim to buy. A whole support industry has grown up around these instruments.
— PatrickThe fake job interview is step one of a supply chain poisoning
Manish is a member of the Rust security response group, and two days after he posted he received a report: someone was using interviews for Rust positions as bait to get candidates to download a library containing obfuscated malicious code, and at least five core community members were targeted. His first read was that this is a supply chain attack — compromise the software maintainer first, then poison hundreds or thousands of companies through a normal-looking update, and eventually reach bank accounts to move money.
— ManishA scammer's best money mule is someone who already fell for it
Scammers go back to people who have already been taken, claim to be investigators working the case, and then offer them a ‘work from home’ job — which is really being a money mule: receive money in your own account under your real name, take a 10% cut, pass the balance on to the next account down the chain. This is the layering step of money laundering. The standard three stages are placement, layering and integration: get the cash into the financial system, create stepping stones that stall anyone tracing it, then buy assets to make it clean.
— PatrickCaller ID and sender addresses can both be forged, so trust only calls you place
The number in your caller ID and the sender address on an email can both be completely spoofed. The one principle you can actually act on: do not trust communication that ‘came to you’; trust only communication you initiate to an endpoint you already know. Bank employees usually have a desk name and an extension, so hang up, dial the official number printed on your card, and ask to be transferred — that verifies them. Even a bank web address in a search ad may be one the scammers bought, which is why you must never call the number in an ad.
— PatrickAnti-fraud should not rest on individual detection but on institutional liability
Credit card fraud causes tens of billions of US dollars in losses every year, and yet it has not eaten the middle class and working families alive, because Regulation E established a clear order of liability: a user whose card is used fraudulently pays at most 50 US dollars, banks usually waive even that, and the loss is then pushed outward to merchants by contract. Putting the liability on the stronger institution, rather than hoping every individual can recognize phishing, is the approach that has actually been proven to work.
— PatrickIn their own words · checked verbatim
LLMs are attacking the cost side of doing sophisticated frauds and cons. And we should start expecting more sophisticated cons against ourselves.
Manish1:06
scams and other forms of property crime committed at scale are committed by businesses.
Patrick2:11
compromise of the identity account is equivalent to compromising many of the high value accounts that are in a given person's ecosystem.
Patrick6:24
I think the most useful thing to understand is what is available today is the worst it is ever going to be.
Manish10:37
if you make a product which is available to move $500 of money around arbitrarily, it is also capable of moving $50,000 by just taking the same thing that moves $500 and repeating it 100 times.
Patrick16:52
My first thought is that this is probably a setup for a supply chain attack.
Manish23:03
you can't trust communication that comes to you. You can only trust communication that you initiate to well-known endpoints.
Patrick36:27
the reason it doesn't eat the middle and working class alive is that there is a defined waterfall for the incidence of that credit card fraud.
Patrick45:42
Figures
| Amount involved in the Hong Kong deepfake fraud case | 25 million US dollars | 9:34 |
| Ransomware ransom per machine | 500 US dollars | 20:02 |
| Money mule's cut | 10% | 26:08 |
| Annual credit card fraud losses | tens of billions of US dollars | 45:42 |
| Regulation E deductible | 50 US dollars | 46:42 |
| Year the UK began requiring banks to reimburse defrauded customers | 2024 | 57:46 |
Glossary
- BEC (business email compromise)
- A scam that breaks into an internal mailbox and impersonates an executive to instruct finance to wire money out.
- money mule
- Someone who receives and forwards fraud proceeds through an account in their own real name, taking a cut.
- AML (anti-money laundering)
- The compliance regime under which banks monitor and block illicit money flows.
- Regulation E
- The US electronic funds transfer rule under which banks bear most of the liability for fraudulent card use.
- desk name
- The alias a bank employee uses with the outside world, so criminals cannot trace them back.
- supply chain attack
- Compromising a software maintainer, then poisoning downstream users in bulk through an updated release.
How to listen
Security engineers, anti-fraud product managers, bank compliance staff, and anyone who controls large sums of money or personal financial information.
Ad breaks you can skip: 30:01-33:03, 47:21-48:12.