While I'm not happy at having to spend my Monday patching a kajillion machines, I welcome more vulnerability writeups in this vein.

http://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h=...

What's the quickest check to see if sshd, or any other listening process, is vulnerable?

(For example, if "lsof | grep ssl" only shows 0.9.8-ish version numbers, is that a good sign?)

Got a long night ahead :/

This sounds risky to me. I'm afraid attackers would benefit more from this decision than coordinated do-gooders.

How's that for responsible disclosure?

To me it sounds kind of like finding out the fence in your backyard was cut open two years ago. Except in this case the backyard is two thirds of the internet.

If i got that right ALL openssl private keys are now potentially compromised.

I hope vendors push fixes soon, and then I guess I'm busy for a few days regenerating private keys.

Suggestion: big, bold TLDR ("The sky is falling. Check your OpenSSL version right now") with a link on what to do sorted by OS vendor.

Step 1: Here's a command to spit out your OpenSSL version. If it is the following string, go to step 2.

Step 2: Here's how to update your OpenSSL. Here are links to guides on reissuing keys.

Probably OK the whole remediation bit links to a wiki that gets updated as the various vendors push their patches.

My opinion, then and now, is that C and other languages without memory checks are unsuitable for writing secure code. Plainly unsuitable. They need to be restricted to writing a small core system, preferably small enough that it can be checked using formal (proof-based) methods, and all the rest, including all application logic, should be written using managed code (such as C#, Java, or whatever - I have no preference).

This vulnerability is the result of yet another missing bound check. It wasn't discovered by Valgrind or some such tool, since it is not normally triggered - it needs to be triggered maliciously or by a testing protocol which is smart enough to look for it (a very difficult thing to do, as I explained on the original thread).

The fact is that no programmer is good enough to write code which is free from such vulnerabilities. Programmers are, after all, trained and skilled in following the logic of their program. But in languages without bounds checks, that logic can fall away as the computer starts reading or executing raw memory, which is no longer connected to specific variables or lines of code in your program. All non-bounds-checked languages expose multiple levels of the computer to the program, and you are kidding yourself if you think you can handle this better than the OpenSSL team.

We can't end all bugs in software, but we can plug this seemingly endless source of bugs which has been affecting the Internet since the Morris worm. It has now cost us a two-year window in which 70% of our internet traffic was potentially exposed. It will cost us more before we manage to end it.

[0] http://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h=...

http://filippo.io/Heartbleed/

It actually exploit the bug, since it was quite trivial, and echo some memory.

It's written in Go, no more than 100 lines. I'll release code in some time.

The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software.

Yes, while that's true, it's not a "read the whole process' memory" vulnerability which would definitely be cause for panic. The details are subtle:

Can attacker access only 64k of the memory? There is no total of 64 kilobytes limitation to the attack, that limit applies only to a single heartbeat. Attacker can either keep reconnecting or during an active TLS connection keep requesting arbitrary number of 64 kilobyte chunks of memory content until enough secrets are revealed.

The address space of a process is normally far bigger than 64KB, and while the bug does allow an arbitrary number of 64KB reads, it is important to note that the attacker cannot directly control where that 64KB will come from. If you're lucky, you'll get a whole bunch of keys. If you're unlucky, you might get unencrypted data you sent/received, which you would have anyway. If you're really unlucky, you get 64KB of zero bytes every time.

Then there's also the question of knowing exactly what/where the actual secrets are. Encryption keys (should) look like random data, and there's a lot of other random-looking stuff in crypto libraries' state. Even supposing you know that there is a key, of some type, somewhere in a 64KB block of random-looking data, you still need to find where inside that data the key is, what type of key it is, and more importantly, whose traffic it protects before you can do anything malicious.

Without using any privileged information or credentials we were able steal from ourselves the secret keys

It really helps when looking for keys, if you already know what the keys are.

In other words, while this is a cause for concern, it's not anywhere near "everything is wide open", and that is probably the reason why it has remained undiscovered for so long.

Edit: downvotes. Care to explain?

However (in openssl.gyp): https://github.com/joyent/node/blob/master/deps/openssl/open...

It disables the heartbeat with the compile time option due to a workaround for Microsoft's IIS, of all things.

So the affected window for node would have been Sep 11, 2012 to Mar 27, 2013 (based on the commit history).

Edit: and just used it to dump 64K from a known-vulnerable device we control. Got a session cookie. Jeez.

  ./bin/Heartbleed openssl.org:443
  2014/04/08 12:15:44 ([]uint8) {
   00000000  02 00 79 68 65 61 72 74  62 6c 65 65 64 2e 66 69  |..yheartbleed.fi|
   00000010  6c 69 70 70 6f 2e 69 6f  59 45 4c 4c 4f 57 20 53  |lippo.ioYELLOW S|
   00000020  55 42 4d 41 52 49 4e 45  47 69 05 e8 90 a6 60 d6  |UBMARINEGi....`.|
   00000030  b4 18 c3 f0 4a 20 40 3a  ef dd 06 8b 87 32 42 00  |....J @:.....2B.|
   00000040  00 00 10 00 0e 00 00 0b  6f 70 65 6e 73 73 6c 2e  |........openssl.|
   00000050  6f 72 67 00 05 00 05 01  00 00 00 00 00 0a 00 08  |org.............|
   00000060  00 06 00 17 00 18 00 19  00 0b 00 02 01 00 00 0d  |................|
   00000070  00 0a 00 08 04 01 04 03  02 01 02 03 09 14 ce 7c  |...............||
   00000080  6d 0c f5 a0 3b cc 16 aa  3b d4 b1 b8              |m...;...;...|
  }

  2014/04/08 12:15:44 openssl.org:443 - VULNERABLE

Edit: I've posted on the support forum, hopefully they get back to us https://forums.aws.amazon.com/thread.jspa?threadID=149690

So it would be a good idea to change all your passwords to critical services like email and banks, once they have issued new certs and updated their openssl.

I've just shut down the webservers running SSL that I can control. If you are vuln and don't want to build openssl from source and can afford the outage. I'd reccomend to do the same.

OTHERWISE BUILD FROM SOURCE IMMEDIATELY, PATCH, AND GET NEW KEYS!

Let's hope CA's don't get swamped by all the CSR's. Or rather let's hope they do so we see people are doing something...

For me right now these are just my hobby projects. So I don't care if they're down. But I imagine it will be fun tomorrow.

And when it's fixed, get new keys.

Btw: I'm a dev. Not a sysadmin though :P

Edit: Debian is patched. I'm online again \o/

TLS heartbeat consists of a request packet including a payload; the other side reads and sends a response containing the same payload (plus some other padding).

In the code that handles TLS heartbeat requests, the payload size is read from the packet controlled by the attacker:

  n2s(p, payload);
  pl = p;

pl is the pointer to the actual payload in the request packet.

Then the response packet is constructed:

  /* Enter response type, length and copy payload */
  *bp++ = TLS1_HB_RESPONSE;
  s2n(payload, bp);
  memcpy(bp, pl, payload);

The bug is that the payload length is never actually checked against the size of the request packet. Therefore, the memcpy() can read arbitrary data beyond the storage location of the request by sending an arbitrary payload length (up to 64K) and an undersized payload.

I find it hard to believe that the OpenSSL code does not have any better abstraction for handling streams of bytes; if the packets were represented as a (pointer, length) pair with simple wrapper functions to copy from one stream to another, this bug could have been avoided. C makes this sort of bug easy to write, but careful API design would make it much harder to do by accident.

Incidentally, someone on the mailing list brought up the issue of having a compiler flag to disable bounds checking. However, the Rust authors were strictly against it.

I don't know C well - why write 19 like this?

But it's a disingenuous one. It ignores the realities of systems. The reality is that there is currently no widely available memory-safe language that is usable for something like OpenSSL. .NET and Java (and all the languages running on top of them) are not an option, as they are not everywhere and/or are not callable from other languages. Go could be a good candidate, but without proper dynamic linking it cannot serve as a library callable from other languages either. Rust has a lot of promise, but even now it keeps changing every other week, so it will be years before it can even be considered for something like this.

Additionally, although the parsing portions of OpenSSL need not deal with the hardware directly, the crypto portions do. So your memory-safe language needs some first-class escape hatch to unsafe code. A few of them do have this, others not so much.

It's fun to say C is inadequate, but the space it occupies does not have many competitors. That needs to change first.

I vehemently disagree. Well-written C is very easy to audit. Much much moreso than languages like C# and Java, where something I could do with 200 lines in a single C source file requires 5 different classes in 5 different files. The problem with C is that a lot of people don't write it well.

Have you looked at the OpenSSL source? It's an ungodly f-cking disaster: it's very very difficult to understand and audit. THAT, I think, is the problem. BIND, the DNS server, used to have huge security issues all the time. They did a ground-up rewrite for version 9, and that by and large solved the problem: you don't read about BIND vulnerabilities that often anymore.

OpenSSL is the new BIND; and we desperately need it to be fixed.

(If I'm wrong about BIND, please correct me, but AFICS the only non-DOS vulnerability they've had since version 9 is CVE-2008-0122)

> but we can plug this seemingly endless source of bugs which has been affecting the Internet since the Morris worm.

If we're playing the blame game, blame the x86 architecture, not the C language. If x86 stacks grew up in memory (that is, from lower to higher addresses), almost all "stack smashing" attacks would be impossible, and a whole lot of big security bugs over the last 20 years could never have happened.

(The SSL bug is not a stack-smashing attack, but several of the exploits leveraged by the Morris worm were)

As to CAs, there have been enough compromises already from other causes that serious crypto geeks like Moxie Marlinspike are trying to change the trust model to minimize the consequences --- see http://tack.io

1. Open a terminal[cd into] /etc/path_to_ssl_certs_folder[per site].

Ex. /etc/ssl/nginx

2. Regen the certs [example nginx mail server]

openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:4096 -keyout mailkey.pem -out mailcert.pem

[this command generates a private key and server cert and outputs to pem's] [Note also the key sizes are 4096, you may want 2048. AND I use -sha256, as sha1 is considered too weak nowadays. These certs are valid for 3650 days...10 years]

Since the command overwrites certs/keys in the current directory of the same name as the outfiles...that's it...you're done. Just restart nginx.

If you change a self-signed cert, like above, expect a new warning from the client on the next connection...this is just your new cert being encountered. Click permantly accept..blah blah.

------------------------------------------------------------------------

On a Windows box:

1. open an admin cmd window and run 'mmc'.

2. Add a new snap-in for Certificates as local machine.

3. Find and 'Disable all purposes for this cert'.

4. Import your new certs from your 3rd party or that you rolled yourself from your enterprise CA.

5. Test new cert.

6. Delete old cert.

[If you run your own CA, you should already know what to do...]

To me, this implies that it's not too easy to exploit, or we would've seen it fixed much sooner.

I just looked at one of my running apache processes, it only has 3MB of heap mapped (looked at /proc/12345/maps). That's not a whole lot of space to hide the keys in.

Supposing you do hit the lottery and get a key somewhere in your packet, you now have to find the starting byte for it, which means having data to attempt to decrypt it with. However, now you get bit by the fact that you don't have any privileged information or credentials, so you have no idea where decryptable information lives.

Assuming you are even able to intercept some traffic that's encrypted, you now have to try every word-aligned 256B(?) string of data you collected from the server, and hope you can decrypt the data. The amount of storage and processing time for this is already ridiculous, since you have to manually check if the data looks "good" or not.

The odds of all of these things lining up is infinitesimal for anything worth being worried about (banks, credit cards, etc.), so the effort involved far outweighs the payoffs (you only get 1 person's information after all of that). This is especially true when compared with traditional means of collecting this data through more generic viruses and social engineering.

So, while I'll be updating my personal systems, I'm not going to jump on to the "the sky is falling" train just yet, until someone can give a good example of how this could be practically exploited.