Skip to content(if available)orjump to list(if available)

Tailscale ate my network (and I love it)

hlieberman

I've never used Tailscale, but I want to highlight specifically for working with AWS, you could consider using AWS Systems Manager to access machines that are in private VPCs[1][2]. This has the advantage of reusing the same credentials used already for AWS, as well as being able to further restrict exactly what can be done with them.

[1]: https://aws.amazon.com/premiumsupport/knowledge-center/syste... [2]: https://medium.com/hackernoon/ditch-your-ssh-keys-and-enable...

metadat

Tailscale is insanely easy, simple, and pleasant to setup/manage compared to AWS anything. The thought of using a cloud-specific solution is kind of disgusting in comparison.

If Tailscale-the-product ever goes rogue or evil, I can always self-host wg or a full on tailscale-equivalent mesh myself. I sleep well knowing this.

gzer0

https://github.com/juanfont/headscale

If you were looking for an open source, self-hosted implementation of the Tailscale control server (as far as I know, that is the only portion of tailscale that tailscale keeps proprietary, and this is the best open source implementation of it).

Edit: wow, this project has really grown from when I last saw it. It is able to configure the vast majority of tailscales base featureset such as ACLs, magic DNS, taildrop file sharing, and so much more. Incredible.

369548684892826

Thanks for pointing this out, going to give it a whirl! Does this solve the thing about having to login using one of Google, Microsoft or Github accounts?

Shish2k

> as far as I know, that is the only portion of tailscale that tailscale keeps proprietary

Also the iOS client, which is hard-coded to only use the first-party control plane, so you can’t use headscale if you have iOS clients :(

metadat

Shower thought:

If I didn't have the goal of buying a house in the Bay Area, I would totally try to find a job with Tailscale.

xcambar

If you have to take a job you don't like to be able to afford the house, be aware that nothing changes once you own the house. Most likely you'll still need the job to afford the running costs of the house.

There's no "I'm free once I'm a home owner" thing.

somedudetbh

I keep catching myself thinking this too.

In general, I think it's kind of an interesting heuristic to think about every so often: right now, putting aside practicalities like my current job or where I live, what looks like a cool place to work, even if the specific role was just taking out the garbage.

Right now the answer is "tailscale" and "oxide" for me.

aaaaaaaaata

Aren't houses in the Bay Area bound to become lower price in the future due to new building being allowed, or subsidies?

Seems like a risky place to invest at this stage.

pid-1

If your sole use case for wireguard is "I need ssh access to a fleet of EC2 machines" then System Manager is definitely cheaper and easier to setup. Even more so if one is using IaC.

p_l

There's even a mostly feature complete open source control +DERP server on headscale

metadat

Thanks for mentioning it by name. Yeah, headscale is very badass.

2Gkashmiri

why dont you use zerotier?

linsomniac

I really like ZeroTier, but the nail in the coffin for me has been that there is no ability to self host a controller, while also using the management web GUI.

Their sales team, when I asked about self-hosing a controller, said it's not necessary because they've never had all the hosted controllers go down, but when I asked about a tweet they sent in May 2020 about their controllers being down, I never got a reply. [1]

My plan was to put ZeroTier on all of our machines and use it as an overlay network that all traffic goes over. But I don't want to open the availability of our network to depending on an external service.

I've all but decided on Nebula, just need to get the deployment worked out. I'm playing with Tailscale right now, and am very impressed. It does have the ability to require MFA on logins that we would like for user VPNs, while still being able to have servers self-authenticate (we respin half of our dev/stg environment every night).

[1] https://twitter.com/ZeroTier/status/1389766385480372225?s=20

rainsford

Can't speak for the parent commentator, but I gave zerotier a try and ended up dumping it mostly because it was unusably slow on single core Linux VMs, making it not a viable option for connecting lower tier cloud VM options. I believe this is on the list of things they're fixing with their next major version, but that version has been very slow in coming out.

inshadows

robocat

Are you implying they were shilling? Accusing others of shilling is against HN guidelines: https://news.ycombinator.com/newsguidelines.html

boredtofears

its not really that hard to setup amazon system manager. like maybe 2-3 hours of overhead.

the idea that you'd pay a subscription fee greater than aws just to avoid a few hours of learning how to set something up is kinda disgusting in comparison.

LilBytes

Resolving to a hostname in n AWS using a tag is eirksome though. MagicDNS on the other hand in Tailscale makes it easier.

FWIW I use both SSM Port Forwarding and Tailscale but Tailscale is easier for both tech and non-tech users IMO.

aranelsurion

I might be misremembering this, but IIRC if your nodes are running on latest Amazon Linux, they already have the ssm-agent pre-installed.

At most you'd need to add some permissions to the instance's role, maybe.

infinityio

How many devices do you need to keep connected? I'm surprised the free tier wouldn't work for you to be honest

pistoriusp

Why do you care what other people do? Just focus on yourself.

throwaway290

> Tailscale is insanely easy, simple, and pleasant to setup/manage compared to AWS anything.

Seriously? What equivalents does it have for Route53 and tons of other services?

I can't tell if this is misleading or pointless.

michelledepeil

The post you're responding to wasn't arguing for capabilities, it was arguing for ease-of-use and it was right.

Your comment is pointless and a straw-man argument.

Hawxy

There's also a new port forwarding parameter on session manager so you can easily connect to a private RDS via a bastion host.

https://aws.amazon.com/about-aws/whats-new/2022/05/aws-syste... https://docs.aws.amazon.com/systems-manager/latest/userguide...

nighthawk454

Oh wow, I’m still using ssh tunnels through the bastion. Gonna try this next week, thanks!

jvolkman

We still need bastion hosts to connect to RDS, but they're no longer accessible from the internet since we use SSM.

`ssh aws-bastion` invokes a ProxyCommand that:

1) finds an actual bastion host by tag using ec2 describe-instances

2) generates a short-lived ssh key and adds it to the ssh agent

3) sends the temporary key to the bastion using ec2-instance-connect send-ssh-public-key

4) starts the SSM/ssh session using AWS-StartSSHSession

Since it's SSH, we can port forward, use multiplexing, etc. We use Google as the IdP, via AWS SSO. Bastions periodically sync users from a Google group.

paranoidrobot

Someone on Reddit posted a shell script[1] that does all the connection setup, key-sending, etc that's a useful base to work from.

[1] https://www.reddit.com/r/aws/comments/df6uip/ssm_tunnelling_...

CubsFan1060

jvolkman

It still requires the bastion host though, right? In which case I'd prefer to just use ssh since it's more familiar.

Also, intellij's built-in database tools support ssh tunneling, it 'just works' with the ProxyCommand method.

moltar

Why do you need steps 2 and 3? I’m able to use SSM with agent without any SSH keys. Also proxying to RDS. Only AWS credentials are required.

jvolkman

Yeah, SSM supports just opening a terminal on the remote host without ssh. We started with regular ssh bastions though so I just stuck with that. They're just no longer routeable from outside.

null

[deleted]

benmanns

FYI when AWS had their little oopsie with the EC2 management APIs not too long ago, I couldn’t connect to an instance via Systems Manager. I also couldn’t adjust my security groups to enable SSH. I also couldn’t shut down the instance. AWS refused to provide any credits as their SLA only covers when the instances are not publicly accessible.

vosper

I’ve used Twingate for this and found it very easy to setup and use. No idea whether experts consider it a good option.

danenania

You can also use AWS Client VPN. It’s not as user-friendly as tailscale, and probably more expensive(?), but it works well enough. It’s also pretty easy to provision with IaC—not sure if that’s the case for tailscale?

thedougd

I'm currently evaluating Tailscale and other VPN solutions as AWS Client VPN doesn't look like it's going to work for us.

AWS Client VPN does not offer any means of unattended configuration or mass distribution. All they offer is a self service portal from which you can download the installable, the profile, and directions. Each use has to manually import the profile into their client.

I'm stunned they're missing this as it would prevent any sizable organization from adopting it.

danenania

Yeah it’s not ideal. It’s nice not to have to run any kind of agent, and to use IAM for access control (not that IAM is so wonderful, but it plugs in easily). I agree though it could be much smoother.

digianarchist

We're using AWS SSM to allow access from a developer's machine to a locked down k8s cluster in AWS.

It works but performance is awful. Slow connection times and pushing images has slow throughput.

atonse

Huge fan of Tailscale here too. They solved every single complaint I had with using WireGuard (provisioning, key exchange, IP assignment, ACLs, etc.) and did it in a splendid and elegant tool that just disappears.

One of the few products I recommend enthusiastically.

To be clear, WireGuard seemed to have the right level of abstraction as a tool for others to build on (just like it built on top of the noise framework), and someone like Tailscale ran with it.

colordrops

The main reason I use wireguard on my personal network is to keep 3rd party clouds out of it. My understanding is that a closed source node that phones home is part of their solution. For my personal case, this is a total no-go.

Operyl

It's worth noting that the Tailscale client is, almost entirely, open source. It's just the iOS/macOS/Windows client code that's closed source (just GUI wrapping it), iirc[0]. The DERP code is also open source.

Additionally, there's an open source reimplementation of the control plane called headscale, as well. The Tailscale team has complimented it, but of course it's all on your own if you choose to run it.[1]

[0] https://github.com/tailscale/tailscale

[1] https://github.com/juanfont/headscale

EDIT: Android code is actually entirely open, oops!

ngcc_hk

Wonder why those are closed source if just gui wrapping.

thejosh

Wireguard is absolutely fantastic for a small network, when you know what you're doing. It's dead easy to setup and configure, you forward some ports, and boom you have a shared network.

I couldn't believe how easy it is to configure. I set it up for work, do a key rotation on a schedule and it's great.

Tailscale I use for home use on my personal laptop + machines and it's fantastic as I don't need to port forward or anything. I changed over from using Wireguard with minimal effort, just changing some IPs over. Probably end up using MagicDNS (their DNS solution), then if I need to change off I'll just change this.

Their free tier is very generous for a single user.

pkulak

My problem is that Wireguard alone works too well. Years ago I got it all set up, forwarded the one port, and it’s been perfect ever since. I really want an excuse to try Tailscale. Some day. Haha

null

[deleted]

onelovetwo

Is this a replacement for something like Mullvad?

linsomniac

No. Mullvad is VPN to the Internet, Tailscale is a VPN among your own machines.

skybrian

It seems like any code running in your browser or on your local machine has access to your home network, which was always true, but now your "home network" includes machines in multiple locations, including AWS.

xlii

That’s a very interesting perspective and I wonder if that will change attack vector.

One reason to have gateways etc. was to ensure that gateway couldn’t be taken over by software installs etc. Access was inconvenient but it was somewhat by design.

Maybe due to this attack vector shifted from directed access to automatic scanning of ever expanding vulnerabilities.

Now, as services stops being accessible from external networks once again they can be accessed in convenient ways thus bringing the old vectors back. Sometimes even giving false sense of security.

wmf

Tailscale has ACLs and then beyond VPNs there's zero trust and BeyondCorp.

moontear

Shoutout to ZeroTier, Nebula which essentially do the same thing. Or Netmaker if you wanna go complete open source/self hosted.

rcarmo

I tried ZeroTier in the past, and it was great, but didn't have an iOS app. Do any of those do?

djchen

ZeroTier has an iOS app now.

jbverschoor

Zerotier works at a lower level. This means you can do other protocols than tcp/ip

anotherhue

I don't think that's correct, they both expose TUN/TAP devices so should be equivalent in their encapsulation abilities. I use both regularly.

https://en.wikipedia.org/wiki/TUN/TAP

FL410

It is correct because a TUN interface is not the same as a TAP interface. The latter is layer 2.

akvadrako

Tailscale can do UDP also.

linsomniac

"TCP/IP" means TCP, IP, and UDP.

ZeroTier can do other protocols like NetBIOS, so you can use it to run LAN games that don't use TCP/IP.

Zizizizz

Yeah netmaker is really easy to set up

mfrw

I organically grew my tailscale network and with the recent `tailscale ssh`[0] it has turned my life around. I have no open ports to anything & be it my personal machine in the depths of my closet or stuff on the cloud; everything is seamless connected.

[0]: https://tailscale.com/tailscale-ssh/

dawson

For those still using SSH as normal, you can setup Tailscale to accept connections only from Tailscale, and ignore any public internet traffic i.e., restrict ssh access to be only over Tailscale. For example, with UFW you could delete every rule except for the “Anywhere on tailscale0” and “41641/udp” rules.

mfrw

Exactly how it `turned my life around`.

I can now go to sleep without having to worry about random bots trying to mine crypto on my machines. To add to the goodness, one does not have to worry about either SSH-keys or remember cryptic passwords.

withinboredom

This seems like a ... strange ... comment. Do you usually worry about your ssh keys getting stolen? How do you use git?

FWIW, setting up something like tailscale is remarkably simple (I'm using PHP here to keep it simple):

In sshd_config:

    AuthorizedKeysCommand /auth_ssh %u
    AuthorizedKeysCommandUser nobody
And in auth_ssh, verify that the user is allowed to connect to that server, then look it up on github (my public keys: https://github.com/withinboredom.keys).

If you want to allow any github user you allow to connect various permissions, check out libnss-ato.

These are all 1 or 2 lines of configuration and are not hard. You just have to know they exist.

tssva

I really want to enable tailscale ssh but I often need to ssh using my phone and so far none of the android ssh clients I have tried work properly with tailscale ssh.

I believe they are all based upon variations of the same java ssh library and exhibit the same behavior. They all connect to tailscale ssh using 'none' authentication but after connecting don't display anything which means I can't get the URL tailscale ssh presents to do its authentication.

Edit: I was just able to work around the issue by installing Termux and using openssh in that environment to do my initial ssh authentication. Afterwards my normal ssh app works.

ptomato

on iOS, at least, the Tailscale app will pop up a push notification for you to authenticate in that case; I'm not sure if the same is true on Android but could be worth checking your notification settings.

tssva

Notifications are enabled for Tailscale but I don't get one to authenticate.

I was just able to work around the issue by installing Termux which provides a small Linux environment on your phone. I was able to use openssh in the Termux environment to connect and get the authentication URL. After that my preferred ssh app can connect without issue.

janci

Tailscale seems like a great product however I do not want 3rd party to be able to add a key to my ACL. Running a custom control plane server is possible, but then there is little benefit for me compared to direct wireguard with a central peer on a VPS. If it would be possible to use just the NAT traversal without key management, that would be it!

Curretly I am running a tiny VPS as a wireguard server, but I do not trust it to be part of my network. Therfore I run one wireguard tunnel to be able to access my router (has no public ip) and second tunnel inside the first to connect through the router to my home network.

Theoretically, it should be possi le with single wireguard tunnel if I set a route to home router via wireguard gateway - but I never managed to make wireguard encrypt a packet if it came from the same wg interface. Can anybody help?

imdsm

I think Tailscale have the right approach by knowing their customer — someone who is happy to have a trusted 3rd party administer parts of their VPN in return for time and cost saving. There are a few here who can't have that, so they instead invest their time into a custom setup with WireGuard which is fine, but for those of us who don't require that level of assurance (there are bigger attack vectors to worry about), Tailscale is fantastic. Quick, easy, and mostly works out of the box.

fancyfish

Tailscale has been a godsend for my team, saving us quite a bit of effort with VPN/firewall administration. There are very few rough edges, and it tends to just work (at least at our scale of a few thousand nodes). We moved over about 8 months ago and have had no issues since. I’ve also moved my home network (RPis, NAS, etc) to their free tier so I can access it remotely.

Some features that are basically effortless and made me choose it over WireGuard and other VPN solutions: easy provisioning, key exchange, IP assignment, ACLs

simongray

I recently set up Tailscale, but unfortunately the phone app leaves a lot to be desired battery-wise (it takes up 30% of my total battery usage) so I think I'll be looking elsewhere.

Initially, I had tried setting up Nebula, but I am unable to get a static IP address for the beacon (a requirement for any of these mesh VPNs), hence why I went with Tailscale which acts as a beacon for you. I think I'll try ZeroTier next.

yoru-sulfur

30% seems way _way_ higher than I would expect.

I've been running Tailscale on all my devices for a couple months now, and I haven't noticed any impact on battery life. I just checked my phone (Android) and it's reporting 1% usage.

I would report an issue if you're seeing numbers that high.

simongray

I use an iPhone. If you google it, you will notice that lots of people have this issue and have had for a long time.

- https://www.reddit.com/r/Tailscale/comments/pm49t9/tailscale...

- https://forum.tailscale.com/t/high-battery-usage-in-ios/1152

- https://github.com/tailscale/tailscale/issues/3363

sph

Weird, 4% on my iPhone. It's always on, though I don't make a lot of connections through it

metadat

I only activate it when I actually need to use it. I don't like running any phone apps in the background, they always suck juice.

oynqr

I had this issue as well, it turned out that something was multicasting frequently, which gets turned into a broadcast because of wireguard's nature. I just switched to plain wireguard with a VPS and my issues magically disappeared.

Zizizizz

You can use netmaker and the native wireguard app

metadat

I love tailscale, too. Also, I read this article before it had any upvotes and learned absolutely nothing new or insightful. Wish the author had kept going.

smackeyacky

It was news to me that Tailscale allowed DNS lookups for particular domains to hit specific resolvers and those resolvers would serve up the internal VPC address, so you need nothing except a subnet router inside the VPC to be working against your secured AWS resources.

It means that you can close all the open ports on your VPC security groups without changing the configuration of how your external systems access the internal AWS services.

It was probably obvious to everybody else, but after I worked that out, Tailscale became my network.

metadat

Magic DNS is a fucking nightmare behind the scenes, I just use raw dog IPs to KISS. The TS app makes it easy to copy to clipboard.

See: https://lobste.rs/s/v4obi8/how_does_tailscale_s_magicdns_wor...

I also set custom hosts file entries for non-mobile devices and kablam, no magic needed (for personal use, anyhow).

aidos

Behind the scenes they’ve obviously got a lot of crazy to deal with, but it seems to work well from the outside (just using it for Tailscale lookups, at least).

Have you had issues with it otherwise?

EDIT actually I do have one gotcha. There’s a switch in the admin panel to override real dns. In theory if you changed that option and your machines were currently using private dns on route53 to find each other you might be in trouble (don’t ask me how I know).

bigiain

It’s not DNS

There’s no way it’s DNS

It was DNS

The only thing worst than “It was DNS”, is “It was DNS, but in this rare and weird edge case only so it never showed up when you tried to debug it”…

I mean, I’m impressed by that capability. But I’m horrified by the potential future support implications. Who’d want to be debugging a problem with “magic DNS” at 2am on a Sunday morning while Prod is down and the entire C suite is half drunk, tired and angry, and breathing down your neck?

dvzk

Sadly almost none of that is complex or surprising if you’re used to dealing with DNS deployments on Linux or BSD. What’s new is bundling a custom system name resolver for machines that can’t forward matching domain requests to specific name servers. Users are often left to their own solutions if they don’t use systemd-resolved or NSS.

null

[deleted]

unnouinceput

Quote: "In the "before tailscale" times, if I needed to test against the production AWS resources or connect dBeaver for database maintenance, I would edit the security group to add my IP address, do my testing, edit the security group to remove myself. This is as error prone as it sounds. I quite often forgot to remove my IP address from the allowed addresses, a major potential security risk when you are travelling."

My takeaway from this is that the author was either lazy or lacked the knowledge to create an automation script that could've done that automatically (the add/remove) based on location. If that's the whole reason for this tailscale praise, kinda of takes away the tailscail actual usefulness and why it exists in first place.

smackeyacky

Author here. Yep, I'm lazy and a cheapskate, so using the built-in AWS solution I thought was too expensive for something I only did occasionally. 99% of the time I'm hooked up to my test system rather than production (as it should be).

madjam002

To add to the OP’s article, Tailscale can map IPv4 to IPv6 addresses when using subnet routers.

Imo this is incredibly handy, as if I want to expose a device to my Tailscale network, I don’t want to have to think about finding an IP address range that won’t conflict with the various local network ranges that my Tailscale devices are on. Especially if you’re using Tailscale in various corporate environments where 10.0.0.0/8 is used a lot.

Now I can just expose e.g 192.168.122.0/24 to my Tailscale network but it’s exposed as a unique IPv6 /120 prefix.

https://tailscale.com/kb/1201/4via6-subnets/

aborsy

How about the old solution of devices connecting to an access VPN running on a nearby AWS VPS?

I have my private network right now. As a plus, devices can make direct connection when they are in restrictive corporate networks (allowing only 443/tcp). Less third parties involved. Seems more secure for personal use.

Sure, it’s not a mesh network, but that doesn’t matter if VPS and devices are in the same region.

But I get that mesh VPN products can be valuable to small businesses: ease of use, ACLs, SSO, central management.

dvzk

Custom peer to peer networks over Wireguard are theoretically more secure. A list of additional potential adversaries who can remotely access your Tailscale network without client manipulation:

- Tailscale’s coordination server

- Tailscale’s cloud provider

- anyone who compromises the coordination server

- SSO provider you have chosen

- SSO’s cloud provider

- anyone who compromises the SSO provider

smackeyacky

There is nothing wrong with a direct VPN, but Tailscale makes them look hard to configure and you don't get the magical DNS stuff they provide.