Get the top HN stories in your inbox every day.
franky47
tinco
This is a nitpick and not pertinent to anything, but base64 encoding your password could make it stronger. Password strength is not just about entropy, high entropy is simply the most effective way of making your password stronger. If your password is 100% randomly generated (as it should be) then base64 encoding it won't do anything.
If however your password is engineered to be easier to remember, for example by using dictionaries or some kind of scheme that has a lower entropy, then the base64 encoding step adds a single bit of strength to your password. Meaning anyone who is brute forcing your password using a smart password cracker, has to configure that cracker to also consider base64 encoding as a scheme, basically forcing it to perform a single extra operation for every try.
Anyway, useless information, you shouldn't be using password schemes like that. The horse shoe battery staple type password style should be quite sufficient I think.
giancarlostoro
> Anyway, useless information, you shouldn't be using password schemes like that. The horse shoe battery staple type password style should be quite sufficient I think.
I wonder if its better to make an encoder that uses words and the output looks like "horse shoe battery staple" except you don't release your dictionary list of potential words output by the encoder, but then you guarantee that you can always re-create a password if you lose it, assuming you don't lose the dictionary file.
starttoaster
I feel like we're discussing the 24 word mnemonic private keys used by crypto wallets with extra steps.
franky47
One could argue that base64 having a shorter output length than its input would weaken any given password, assuming a brute force attack (not a dictionary one).
LelouBil
Base64 turns 6 bits to a ASCII character (8 bits), I don't think it's possible to have a shorter output length.
luismedel
Sorry, but it's the opposite.
A base64-encoded string will be ~30% longer than the original string.
osigurdson
I would think every CS grad would know the difference between these things. Everyone interested in coding could learn these concepts in an afternoon.
xp84
You and I would both think -- but on the other hand, I've interviewed a lot of people who have dispossessed me of that illusion. Seemingly simple concepts don't always stick.
CobrastanJorji
A related thing always worth emphasizing: hashes aren't necessary cryptographically secure!
Hashing has many purposes besides security, and for that reason there are many hash libraries. If you plan on using hashes for something related to security or cryptography, you need to use a hash designed for that purpose! Yes, CRC hashing is really fast, and that's great, but it's not great when you use it for user passwords.
remram
Password-masking functions are not usually what's referred to as "cryptographically-secure hash", e.g. unsalted SHA-2 is one but not the other. For example resistance to enumeration or rainbow tables is not a requirement for cryptographic hash functions, but is important for good password-hashing functions.
fbdab103
Unless you have really specific requirements (hashes/second, hash size no bigger than X characters, etc) is there any reason not to default to sha256?
I still see newly released projects that choose md5. Like, sure, for the intended use case, probably nobody will construct a collision, but why even allow the possibility?
franky47
It's not much about collisions, more like predictability of the output. You can trivially construct a rainbow table of the most common N passwords and test a dump of SHA-256 hashes against it. Also, SHA-256 is vulnerable to length extension attacks, so it may not be suitable in some applications on variable-length inputs.
Generally speaking, hashing user-provided data as-is is only a guarantee of integrity, not of authenticity (see HMAC), nor secrecy.
globular-toast
This is why computer science as a discipline matters.
throw0101a
> Many junior devs have been bitten by not being told early enough the difference between encryption (requires a secret to be reversed), hashing (cannot be reversed) and encoding (can always be trivially reversed).
If they want to encrypt something just tell them to use ROT13 twice.
xp84
This is dangerous and outdated advice. ROT13 should always be used an odd number of times to avoid CVE-2022-13!
undefined
sirk390
I really though you were going to say "Chat-GPT detected" when quoting that message.
p4bl0
A funny thing about Base64: if you iterate encoding starting from any string, a growing prefix of the result tends towards a fixed point. In Bash:
$ function iter {
N="$1"
CMD="$2"
STATE=$(cat)
for i in $(seq 1 $N); do
STATE=$(echo -n $STATE | $CMD)
done
cat <<EOF
$STATE
EOF
}
$ echo "HN" | iter 20 base64 | head -1
Vm0wd2QyUXlVWGxWV0d4V1YwZDRWMVl3WkRSV01WbDNXa1JTVjAxV2JETlhhMUpUVmpBeFYySkVU
$ echo "Hello Hacker News" | iter 20 base64 | head -1
Vm0wd2QyUXlVWGxWV0d4V1YwZDRWMVl3WkRSV01WbDNXa1JTVjAxV2JETlhhMUpUVmpBeFYySkVU
$ echo "Bonjour Hacker News" | iter 20 base64 | head -1
Vm0wd2QyUXlVWGxWV0d4V1YwZDRWMVl3WkRSV01WbDNXa1JTVjAxV2JETlhhMUpUVmpBeFYySkVU
[1] https://twitter.com/p4bl0/status/298900842076045312
[2] https://news.ycombinator.com/item?id=5181256
[3] https://www.reddit.com/r/compsci/comments/18234a/the_base64_...
[4] https://web.archive.org/web/20130315082932/http://fmota.eu/b...
undefined
jstanley
Whoa, that's really neat!
preciousoo
Aww the article linked in Reddit is dead
p4bl0
As explained in my comment above, the blog is dead, but link 4 in my comment is a backup of this article by archive.org.
preciousoo
Oops missed that
benjaminwai
Just a note with the Bash encoding method. It should be with the -n option:
$ echo -n "abcde" |base64
meindnoch
Simply don't use echo if you want predictable output. Use printf. https://linux.die.net/man/1/printf
ndsipa_pomu
This is the way
(however, the parent's use of "echo" would be fine as it's not using a variable and so won't be interpreting a dash as an extra option etc)
gdavisson
echo -n is not safe, because some versions of echo will just print "-n" as part of their output (and add a newline at the end, as usual). In fact, XSI-compliant implementations are required to do this (and the same for anything else you try to pass as an option to echo). According to the POSIX standard[1], "If the first operand is -n, or if any of the operands contain a <backslash> character, the results are implementation-defined."
[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/e...
quesera
Not all echos accept -n to suppress newlines.
printf is always the better choice.
manojlds
Yeah been bit by few times. Somehow keep forgetting.
adamzochowski
There is also base64URL , where the encoding uses different ascii characters that are url safe. I have seen some devs use BASE64URL but call it just base64, and that can lead to some problems for unaware.
JimDabell
The problem with base64url is that ~ and . are not letters. This means that double-clicking on something encoded with base64url isn’t going to select the whole thing if you want to copy-paste it. This is annoying needless friction in a huge number of use cases. Base62 encoding (0-9A-Za-z) is almost as efficient as base64url and retains URL safety but is more easily copy-pasted.
If you want to eliminate ambiguity for human readers, you can drop to Base58 but in almost all cases, if you are BaseXX-encoding something, it’s long enough that copy-pasting is the norm, so it doesn’t usually matter.
nly
Encoding and decoding base58 is a lot less efficient (needs arithmetic multiplication with a carry across the stream).
Base32 is sufficient in most cases and can avoid some incidental swear words.
If you want density go for Z85, which is a 4 -> 5 byte chunked encoding and therefore much more efficient on a pipelined CPU.
JimDabell
Base32 has almost half the efficiency of Base62; Z85 suffers from the same problem as Base64 in terms of including word-breaking punctuation.
iainmerrick
The problem with base64url is that ~ and . are not letters.
No, typically the extra characters used are “-“ and “_”. That’s what the table in the IETF link shows.
Acinyx
The issue remains: "-" breaks double clicking to select the full string, which means you'll have to manually select all the characters before copying. Same thing happens with UUIDs: using double clicking, you can only select one block at a time.
This isn't a major issue, which means there's no easy answer and it generally comes down to preference if this is a requirement or not.
Aachen
> The problem with base64url is that ~ and . are not letters. This means that double-clicking on something encoded with base64url isn’t going to select the whole thing
Well, you're in luck: tilde and dot aren't part of base64url
Acinyx
The issue remains: "-" breaks double clicking to select the full string, which means you'll have to manually select all the characters before copying. Same thing happens with UUIDs: using double clicking, you can only select one block at a time.
This isn't a major issue, which means there's no easy answer and it generally comes down to preference if this is a requirement or not.
jameshart
Base64url also typically omits padding.
Since a base64 string with padding is always guaranteed to be a multiple of four characters long, if you get a string that is not a multiple of four in length, you can figure out how much padding it should have had, which tells you how to handle the last three bytes of decoding.
Which makes it a little confusing why base64 needs == padding in the first place.
Zamicol
Any time base conversion comes up, I shamelessly plug my arbitrary base converter: https://convert.zamicol.com
The base64 under "useful alphabets" is the "natural", iterative divide by radix, base. There's the RFC's "bucket" conversion base under extras.
BlackFly
If you ever need to encode something and expect people to type it out... I recommend using https://en.wikipedia.org/wiki/Base32 instead. Nothing more frustrating than realizing that (because of bad fonts often) that I was an l or a 1 or that o was an O or was it a 0?
simbyotic
Perhaps a bit pedantic, but would be more accurate to say that Base64 encodes binary data into a subset of ASCII characters, since ASCII has 128 code points - 95 printable characters and 33 control characters - whereas Base64 uses 64 - 65 if we include the padding - of those.
soliton4
it didnt go into detail about the purpose of the = / == padding. and it also didnt show in the example how to handle data that can not be devided into groups of 6 bits without bits left over. i think i have an understanding of how to do it but it would be nice to be certain. could someone address the following 2 questions in a short and exhaustive way:
- when do you use =, when do you use == and do you always add = / == or are there cases where you dont add = / == ?
- how to precisely handle leftover bits. for example the string "5byte". and is there anything to consider when decoding?
tangent128
Your questions are related.
For context: since a base64 character represents 6 bits, every block of three data bytes corresponds to a block of four base64 encoded characters. (83 == 24 == 64)
That means it's often convenient to process base64 data 4 characters at a time. (in the same way that it's often convenient to process hexadecimal data 2 characters at a time)
1) You use = to pad the encoded string to a multiple of 4 characters, adding zero, one, or two as needed to hit the next multiple-of-4.
So, "543210" becomes "543210==", "6543210" becomes "6543210=", and "76543210" doesn't need padding.
(You'll never need three = for padding, since one byte of data already needs at least two base64 characters)
2) Leftover bits should just be set to zero; the decoder can see that there's not enough bits for a full byte and discard them.
3) In almost all modern cases, the padding isn't necessary, it's just convention.
The Wikipedia article is pretty exhaustive: https://en.wikipedia.org/wiki/Base64
pixelbeat__
Padding is only required if concatenating / streaming encoded data. I.e. when there are padding chars _within_ the encoded stream.
Padding chars at the end (of stream / file / string) can be inferred from the length already processed, and thus are not strictly necessary.
Note how padding is treated is quite subtle, and has resulted in interesting variations in handling as discussed at: https://eprint.iacr.org/2022/361.pdf
eatporktoo
from the article: "Every Base64 digit represents 6 bits of data. There are 8 bits in a byte, and the closest common multiple of 8 and 6 is 24. So 24 bits, or 3 bytes, can be represented using four 6-bit Base64 digits."
So you're essentially encoding in groups of 24 bits at a time. Once the data ends, you pad out the remainder of the 24 bits with = instead of A because A represents 000000 as data.
For the record, I had to read the whole thing twice to understand that too.
jameshart
Not quite. The ‘=‘ isn’t strictly padding - it’s the padding marker. You pad the original data with one or two bytes of zeroes. Then you add ‘=‘ to indicate how many such bytes you had to add.
This is because if you’ve only got one of the three bytes you’re going to need, your data looks like this:
XXXXXXXX
XXXXXX XX????
XXXXXX XX0000 000000 000000
The ‘=‘ isn’t just telling you those last 12 bits are zeroes - they’re telling you to ignore the last four bits of the previous digit too.
Similarly with two bytes remaining:
XXXXXXXX YYYYYYYY
XXXXXX XXYYYY YYYY??
XXXXXX XXYYYY YYYY00 000000
Funny side effect of this:
Some base64 decoders will accept a digit right before the padding that isn’t either a multiple of four (with one byte of padding) or of 16 (with two).
They will decode the digit as normal, then discard the lower bits.
That means it’s possible in some decoders for dissimilar base64 strings to decode to the same binary value.
Which can occasionally be a security concern, when base64 strings are checked for equality, rather than their decoded values.
rezmason
Here is my Base64 encoder shader:
https://github.com/Rezmason/excel_97_egg/blob/main/glsl/base...
I got it down to about thirteen lines of GLSL:
https://github.com/Rezmason/excel_97_egg/blob/main/glsl/base...
I use it for Cursed Mode of my side project, which renders the WebGL framebuffer to a Base64-encoded, 640x480 pixel, indexed color BMP, about 15 times per second:
drawkbox
Blast from the past, the Excel easter egg. Solid.
bakkoting
There's some additional interesting details, and a surprising amount of variation in those details, once you start really digging into things.
If the length of your input data isn't exactly a multiple of 3 bytes, then encoding it will use either 2 or 3 base64 characters to encode the final 1 or 2 bytes. Since each base64 character is 6 bits, this means you'll be using either 12 or 18 bits to represent 8 or 16 bits. Which means you have an extra 4 or 2 bits which don't encode anything.
In the RFC, encoders are required to set those bits to 0, but decoders only "MAY" choose to reject input which does not have those set to 0. In practice, nothing rejects those by default, and as far as I know only Ruby, Rust, and Go allow you to fail on such inputs - Python has a "validate" option, but it doesn't validate those bits.
The other major difference is in handling of whitespace and other non-base64 characters. A surprising number of implementations, including Python, allow arbitrary characters in the input, and silently ignore them. That's a problem if you get the alphabet wrong - for example, in Python `base64.standard_b64decode(base64.urlsafe_b64encode(b'\xFF\xFE\xFD\xFC'))` will silently give you the wrong output, rather than an error. Ouch!
Another fun fact is that Ruby's base64 encoder will put linebreaks every 60 characters, which is a wild choice because no standard encoding requires lines that short except PEM, but PEM requires _exactly_ 64 characters per line.
I have a writeup of some of the differences among programming languages and some JavaScript libraries here [1], because I'm working on getting a better base64 added to JS [2].
[1] https://gist.github.com/bakkot/16cae276209da91b652c2cb3f612a...
vikrant17
I am still not convinced with the reason for using base64.
1. "Another common use case is when we have to store or transmit some binary data over the network that's supposed to handle text, or US-ASCII data. This ensures data remains unchanged during transport."
What does it mean by network that handles text? Why should the network bother about the kind of data in the packet. If the receivers end is expecting a binary data, then why is there a need to encode it using base64. Also if data is changed during transport like "bit-flipping" or some corruption, then should't it affect the credibility of the base64 endcoded data as well.
2. "they cannot be misinterpreted by legacy computers and programs unlike characters such as <, >, \n and many others."
My question here is what happens if the legacy computers interpret characters like <,, > incorrectly? If you sent a binary data, isn't that better since its just 0's and 1's and only the program that understands that binary data, will interpret?
bathwaterpizza
Thanks for the read, it's a very simple encoding but I never decided to find out how it works either, good to know.
Get the top HN stories in your inbox every day.
> It's important to remember that we are not encrypting the text here.
Thank you for emphasizing this. Many junior devs have been bitten by not being told early enough the difference between encryption (requires a secret to be reversed), hashing (cannot be reversed) and encoding (can always be trivially reversed).
Also good to know that while the output looks random, it follows the same entropy as the input. Ie: don't base64 encode your password to make it stronger.