I'm afraid I don't have a lot to add to this conversation but I have to say I just love Tailscale. I don't often run across software that feels so right and when I do it's a great surprise. Every time I see a new feature they're releasing I'm always impressed at how adept they are at targeting modern pain points.
I grew up and got into software by messing around with self-hosting web servers and game communities as a kid. As time has gone on I felt like we had lost some of the magic of easily sharing your machines and your creations with other people. We have a ton of services where you can now deploy and share your creations, but we've moved further and further away from direct sharing. There were plenty of good reasons why this has happened, with security being the most obvious factor, but it still makes me a little sad. I want my things to be able to talk to each other no matter where I am. I want to be able to invite my friends in and have access to my stuff.
Tailscale makes all of that quick, easy and awesome. I think it's really neat, makes me feel like a little nerdy kid again.
> I'm afraid I don't have a lot to add to this conversation but I have to say I just love Tailscale.
Strongly seconded. In my last company we used TailScale in some medium-advanced configurations, and from the dead-simple basic stuff up though some of the trickier stuff it's just a joy to use.. It's making much better networking practices highly-accessible and I'd bet ends up making the Internet a more secure, better organized system as a whole.
They run an amazingly transparent engineering process, for example their issue page (https://github.com/tailscale/tailscale/issues) is a model of transparent, responsive, involved open development. They embrace cool, modern, quirky stuff like NixOS (https://tailscale.com/blog/nixos-minecraft/). It's just generally really high-quality software developed with a very cool "hacker" philosophy.
TailScale is IMHO the coolest place to work right now, and something that almost any software company should look at if they do any networking.
If there's anything not to love, I can't see it. :)
Tailscale is cool, but if we focus on the product that this post discusses, Funnel won't give you the ability to use your own domain name. Cloudflare Tunnels will do that though. I will continue to use Tunnels.
That’s great to hear! I’ve been turning TS off and on when accessing services to make it through the day, but as soon as the battery use goes down (to plain wireguard app levels) I’ll be using it for DNS as well. Then it will truly be TS all the things for me.
There's a number of things. (at least four kinda five things you might mean by "keepalives", and the answer is all of them, so yes, and then others.) The same code that runs on iOS and Android also runs on Linux servers (where it was originally developed) where battery or perfect network efficiency wasn't really a top concern. When that code was moved to mobile, a few efforts were made to improve behavior on phones but not enough.
We're also working on measurements to make sure we objectively fix things and don't regress later in the future once it's fixed.
> As time has gone on I felt like we had lost some of the magic of easily sharing your machines and your creations with other people.
> I want my things to be able to talk to each other no matter where I am.
What isn't easy about forwarding packets destined for port 80/443 of your public IP to the local service in question and being a part of the public Internet like things were from the start?
Using Tailscale is the opposite of self-hosting, you're bringing someone else's third party service in, and adding more complexity and another point of failure.
> What isn't easy about forwarding packets destined for port 80/443 of your public IP to the local service in question and being a part of the public Internet like things were from the start?
- Not every home internet service gets a publicly routable IPv4 address anymore (e.g. CGNAT)
- Not every home internet service gets a static IPv4 address so folks have to handle DynDNS
- Not everyone is comfortable exposing their home network IP address in DNS (Tailscale only shares the endpoint IP once the endpoint is auth'd onto the network)
- Not everyone is comfortable configuring heavy auth/fail2ban/app layer safeties (Tailscale makes the services uncontactable unless you are auth'd into the Tailscale network)
- Not everyone is comfortable/can be bothered configuring Wireguard in highly dynamic environments
> Using Tailscale is the opposite of self-hosting, you're bringing someone else's third party service in, and adding more complexity and another point of failure.
Self-hosting need not be a zealot position - rather one can pick and choose what makes sense for them. Tailscale allows you to build your own network where all the nodes are auth'd (and tailscale lock means you don't even need to trust their keys by default) and non-public internet routable but still globally reachable from known safe devices. This can actually make folks more comfortable with self-hosting their own stuff since it removes so many other considerations. There is also headscale if folks want to self-host the coordination server.
Some argue that a third party service adds complexity and a point of failure. I'll point out that configuring a self-hosted publicly exposed thing from scratch for the first time has a rabbit hole of unknown complexity to the uninitiated. A tool like Tailscale can remove some of those complexities allowing focus on others.
>- Not every home internet service gets a static IPv4 address so folks have to handle DynDNS
For anyone who has only this specific problem out of your list, one solution is to get an HE tunnel. It's what I do.
If my ISP ever gets off its ass and implements IPv6 like it promised three years ago, I'll consider using that directly, though its current indication is that the IPv6 addresses will be dynamic for non-business customers which defeats the purpose.
I have gigabit fiber and it's IPv4 only. My ISP blocks incoming ICMP messages so I can't set up a HE tunnel. I used to use Route48, but they shuttered due to abuse, so I don't know what to do anymore.
Wireguard config is few lines (interface addresses, keys, AllowedIPs, post up and down). Simpler than SSH. You can run it on a cloud instance close to users.
Tailscale is still simpler and provides additional features. A small team or startup will appreciate Tailscale’s access controls.
> What isn't easy about forwarding packets destined for port 80/443 of your public IP to the local service in question and being a part of the public Internet like things were from the start?
Most of the evil in the world currently can be traced back to NATs and dynamic IPs.
In a more general sense, I think these compromises were made available because of a consumerist attitude towards the internet. Yes, we had a real issue with ipv4 exhaustion, but it if it affected businesses who couldn’t even host a website anymore, would this really have been an issue still? More likely people said that these things were an ok workaround because consumers don’t need X anyway. Sometimes these smart hacks engineers are so good at coming up with invalidate crucial invariants about the systems we love.
> As time has gone on I felt like we had lost some of the magic of easily sharing your machines and your creations with other people. We have a ton of services where you can now deploy and share your creations, but we've moved further and further away from direct sharing.
This is interesting, as it hasn't been my experience on the hobbyist side (game servers, personal projects, etc). ngrok, localtunnel, tunnelmole, rathole, tunnelto, zrok, et al. If the use case is just sharing something you built thats behind NAT / on a private subnet, there is no shortage of solutions.
I constantly read good things about Tailscale, as well as to a lesser degree Cloudflare, that I think I'm missing out.
But I've experienced so many times that companies change things and this can mess up the workflow or infrastructure really bad, adding days of work to implement an alternative.
With your hype, how much do you trust that you can rely on Tailscale? Should I feel safe when giving them control?
Any company can take a turn for the worse, and any time you've got SaaS deep in your stack there's risk there.
I can only say that I worry about TailScale growing up to be evil the least of basically every SaaS company I've ever used. They seem extremely serious about making the interaction a "win/win" and keeping it that way as they grow.
This feature is a delight to use. I've tested a few web applications, APIs, and webhooks using it over the last month or two and only experienced a handful of glitches even before it was in beta.
I like the idea of consolidating all my network ACLs with a single configuration file with Tailscale, but I don't like being wedded to a proprietary platform for my personal use. Hopefully headscale gets a similar feature, perhaps minus Tailscale's DNS management.
The is like DynDNS on steroids. Awesome job. It should be noted that for high bandwidth applications, you'll incur a bit of a penalty due to hops but other than that it's pretty solid.
Every time I see tailscale do something really neat I'm always a little disappointed to find out they still offer only the three auth schemes- and I really don't want to tie my networking to google/github/ms. On top of the various tinfoil hat reasons, I know a variety of people who have had these accounts terminated out of the blue, and it throwing out my networking stack would be insanely aggravating.
If you're reading tailscale, I will pay you actual real dollars per month to offer a different not-tied-to-a-megacorp authentication scheme. Till then, guess I've got headscale.
Well god damn there it is! Three days fresh, even! Thanks!
Looks like a fair lot of work to get it configured, but few good things come entirely free. Wonder if there's enough people that could get together for a communal one...?
Got to the end of that post and thought: definitely don't want to self host that!
Are there good options for an IdP that has good data policies that are easy to wire in with tailscale? I'm not opposed to paying for it. I wonder if Zoho can do this for me, I'm very happy paying them $12/yr for email.
Question about the docs, it mentions that "The WebFinger endpoint must be hosted at the domain of the email address provided during setup". Would it be possible to support a subdomain?
Also, a small ask: could the webfinger request that's sent include the `rel` and a well-known user resource params, for the situations where there's already a webfinger implementation there that isn't 100% under dev control which requires these params
like
GET /.well-known/webfinger?
resource=tailscale-webfinger%3A%40mydomain.com&
rel=http%3A%2F%2Fopenid.net%2Fspecs%2Fconnect%2F1.0%2Fissuer
HTTP/1.1
Host: mydomain.com
lastly, is this request resent at every auth event?
Are there really Microsoft accounts that were terminated out of the blue? I always had the feeling they acted a bit more responsibly around that than Google.
How is it with high bandwidth application? E.g would it be okay to put my media server behind it? Currently tunneling it through a VPS so cloudflare doesn't get mad.
Tailscalar here: there is a bandwidth limit, it's a funnel, not a hose. We don't announce what the bandwidth limit is, but please keep in mind that it does exist. I would suggest setting up your media server inside your tailnet for the best experiences, but it depends on who you are sharing it with and why.
I might be missing something; isn’t a Tailnet a bunch of user devices with wireguard tunnels connecting to each other directly? Where does the limit happen?
(And thanks for your work!)
Edit after 1 minute: of course, limit on Tailscale Funnel itself. (Too deep into thinking about Tailscale and forgot about the actual topic of the post. )
Tailscale internally tries to punch NATs to make connections, but if it fails, it will use relay servers. These connections are throttled, and there'd probably be usage limits at some point.
If you allow tailscale through by using ipv6 or port forwards, tunnels will run at line speed with no limits.
fundamentally, something has to be punching NAT somehow, so they're probably taking the traffic on their own servers and relaying it to your machine via the tailnet.
Hola, how would the bandwidth limit work within the tailnet if I am accessing it from outside my home network? Wouldn't it incur some bandwidth on Tailscale's end?
I wonder if the DERPy-stuff helps remove most of the bandwidth concerns - thinking out loud...
Only the setting up of the session, it's effectively P2P then. Routing traffic back out onto the general internet for people without tailscale in your private net will be b/w limited, as mentioned.
Since tailscaled uses the tun/tap driver and thus copies all traffic to userspace (and back), it is extremely inefficient. On my Haswell i5 (plus multiple servers with comparable hardware) the process consumes 40% of CPU time at just 4 MiB/s, and close to 100% at 10-11 MiB/s (with recent sendmmsg/recvmmsg patches¹).
This is about ~2-3x worse than similar applications written in highly optimized C, so don't expect any miracles from further optimizations unless they switch to kernel Wireguard (which doesn't seem likely in the nearby future).
They claim it's very difficult if not impossible, but this sounds like an issue with their architecture — a similar application from their competitors² has had kernel WireGuard support from the start (no relation, I don't even use it and cannot recommend for or against it).
Tailscalar here, for what it's worth, I run my plex server on Tailscale (i5 10600) and I haven't noticed any observable lag due to the TUN/TAP driver. Even with 4k bluray rips at several tens of megabits per second of video quality. I also regularly get near the limit of gigabit ethernet when transferring big files like machine learning models (the 1280 byte MTU plus WireGuard overhead adds up over time and can make the application observed rate be less than what the NIC is actually doing).
Kernel WireGuard for Tailscale is hard because of DERP (HTTPS/TCP fallback relay, all connections start over DERP so that they can Just Work if hole punching fails), but I'm sure it could happen with the right combination of eBPF and Rust in the kernel. It'd be a bit easier if there was a high level abstraction for using the kernel TLS stack to do outgoing TLS connections.
Isn’t it also a UDP issue in general or at least the way packet switching works in Golang on major OSs? I did a bandwidth benchmark over local network over tailscale vs vanilla (in the 100MB/s ballpark) and tailscale was 10-20% slower and used tons of CPU.
As a baseline I tried pushing blank UDP packets with Golang (on Darwin and Linux) at saturated capacity and it ALSO used similar excess CPU, causing dropped packets. My take at the time was that it was primarily the syscall overhead per packet (vs per arbitrarily sized buffer in TCP), and a lack of efficient OS APIs in Golang. Is there truth to this analysis?
Hi! Tailscaler here, one of the folks who worked on the recent throughput improvements. One of the machines I was testing with during our work on segment offloading was a Haswell. I absolutely understand your concern if we're using 40% of CPU at 4MiB/s, we should be doing substantially better than that on efficiency. In our various testbeds which include CPUs like yours, we see higher performance. If you'd like us to look into the issue, do email support@tailscale.com - we'd be really happy to dig in and find the cause.
We have continued our work on performance improvements, and along that path, as an example, we recently diagnosed an issue with a change in the kernel frequency scaling governor that has a regression that Tailscale can tickle and we have an ongoing discussion with the kernel maintainers about that problem. I'm not at all assuming this particular thing is the key source of the performance you're observing, it is more to provide an anecdote that we're still digging deep into areas where we aren't performing well and finding the root cause, and working both inside and outside to address those and where appropriate to add workarounds as well.
I observe there's about 37% overhead when using TS connection on a local gigabit network.
Copying large file from Synology DS1821+ NAS (Amd Ryzen V1500B) to Windows PC (i7-6700K)
is about 111-113 MB/s when accessing NAS directly and 70-73 MB/s when traffic goes through TS
(different large files, so no caching here).
My back of the napkin math says there should be a 40 byte overhead for wireguard around tailscale 1280 byte packets. That's only about a 3% overhead on the direct wire. What is your testing methodology so I can attempt to replicate it in the lab?
I meant overhead in a broad sense - both packet size and CPU load combined - what end user actually care about.
My test is what I have to do fairly often: use Windows Explorer to copy 70-100gb file
from a network NAS to a local drive. Every so often I click on the wrong network share
pinned in the Explorer and see slow transfer speed.
• Ngrok pulled a pricing bait-and-switch a year ago increasing prices to $240/year/user if you wanted a stable subdomain, even for bandwidth-trivial users.
-
Edit: Looks like they now have an $8/month/user tier for a single stable subdomain and now offer some edge hosting as well.
$8/user/mo is still far too much for a stable domain without the spam-guard intermediary page, and I'm glad there's some free competition in this space now.
This is my first time using tailscale, and I set up and figured out funnel within fifteen minutes.
from what I can gather it provides the same functionality as ngrok without reaching for another tool. If Tailscale already exists in your networking tool belt this functionality comes really handy.
Cloudflare gets a lot of criticism on HN (I can fundamentally understand why) but it turns to irrational blind absolutist hatred very quickly.
Cloudflare tunnels have been around for a while. They have a variety of features (IMO) well beyond what Tailscale has in beta here.
In terms of the other comments, Cloudflare has many millions of satisfied customers and roughly 80% of the CDN market so people hosting internet facing properties obviously see value in what they provide.
Cloudflare tunnels are a more mature, more capable, more performant, and cheaper version of Funnel backed by one of the largest networks in the world with hundreds of other features from CloudFlare tailscale doesn't have (and factoring in network, never will).
If you have some grudge against Cloudflare for MITM, ToS, etc now you have an alternative (of sorts) to Cloudflare tunnels.
From the article: When you turn on Funnel, we create public DNS records for your node.tailnet.ts.net name that points to a set of ingress servers we operate around the world, and then we give those servers very limited access to your tailnet.
The funnel relays do SNI-based routing to the target machine in your tailnet, and that machine does the TLS termination. We use the initial TLS handshake to route the connection, but after that it's just opaque bytes to us. You can verify this in the client's source code, and use CT logs to see that there are no additional issued TLS certs beyond the one your end-machine created.
Yesterday, I set up tailscale on a GCP box for just this - running a local server and serving it on GCP. I thought hmm, wouldn't it be cool if tailscale could just do this for you? And now, it does.. lol. Super cool!
I am really enjoying tailscale (even though I have a long standing issue with their log level!)
I even pay for their service as a business!
However their limits on the number of people named in ACLs seems far too low, if anyone from tailscale is reading, it would be great if ACL limits scaled somewhat with seats, because as it stands we get significantly less secure as we grow.
Theres also a feature for autogroups which would be cool, but seems not fully explored.
I know the big features are shiny and fun and drive a lot of attention to the product, but I hope it doesnt get in the way of being, fundamentally, a solid VPN solution.
Looks very similar to Cloudflare Tunnels, except Tailscale goes out of their way to say web traffic only (CF had a bunch of shenanigans) and you don't need your own domain name. And you get all Tailscale's mesh network functionality too, which is awesome.
No complaints here, I seriously love what they're doing. Been tinkering a bit with it but it's had been a great utility and one that literally just works. Been trying to make inroads at $WORK with it as we use so much extra cruft that needs maintenance, breaks, isn't that performant really, stateful, no exposing or ACL management that doesn't require CA shaped pain.
I was just reading about it the other day, pure ingenuity!
For those who don't have time to read, Tailscale uses a quirk in how stateful firewall treats inbound UDP traffic to allow connection to a remote server without it opening up to the public.
It only opens up to another machine validated by public keys.
It serves similar purpose as opening firewall to just a specific IP/port and dynamically change the IP/port as the other machine moves or disconnects. One of the main advantage is that it works behind NATs you don't control (i.e. public WiFi).
Edit: also most home routers do not have the ability to dynamically open up to specific IPs based on where your outside machine is.
Can someone confirm that this is functionally equivalent to CloudFlare tunnels? I think so, but I'm not 100% sure about it, because I don't fully understand what a Tailscale network does.
Not sure why it's called funnel, as a funnel is something that takes a bigger amount of something, and transforms it into a smaller amount of something.
If you're like me, you might've missed that they have a semi-hidden "Personal Pro" that supports 100 devices, 2 subnet routers, and custom auth periods for $48 per year.
Not really - more like a managed wireguard config system, with fallback VPN for NAT punch though when needed (so it always works, no matter the network). Traffic is direct when it can be, but when it can't it still just works. Nothing that isn't possible manually, but is exponential in effort to maintain as you add systems, made super easy.
Plus nice features are appearing all the time, like file sharing, Funnel, magic DNS, etc,
Pretty much but it makes the experience so much better. Like stable IP/DNS to al of your machines no matters how are those configured/accessed the Internet. Or “air drop” files between machines
I grew up and got into software by messing around with self-hosting web servers and game communities as a kid. As time has gone on I felt like we had lost some of the magic of easily sharing your machines and your creations with other people. We have a ton of services where you can now deploy and share your creations, but we've moved further and further away from direct sharing. There were plenty of good reasons why this has happened, with security being the most obvious factor, but it still makes me a little sad. I want my things to be able to talk to each other no matter where I am. I want to be able to invite my friends in and have access to my stuff.
Tailscale makes all of that quick, easy and awesome. I think it's really neat, makes me feel like a little nerdy kid again.