Machinify

Machinify gets HITRUST with low overhead using Tailscale SSH and ACLs

Headquartered in California, Machinify is a remote-first revolutionary software company with a mission to ensure that patients get the right treatment, at the right time, at the right price. The cloud-based Machinify AI platform delivers products that, bit by bit, are transforming health insurance care and claims administration from a human-powered, error-prone series of workflows fueled by faxes and spreadsheets to a world of transparent, real-time care and payment decisions.

Machinify’s infrastructure includes between 200 and 2000 AWS instances, with a churn rate of up to ten iterations per day. Machinify’s developers need a way to access these instances in a way that’s secure and easy to manage, which proved to be a challenge to implement.

Their old VPN solution was a whole lot of work

Before turning to Tailscale, Machinify used OpenVPN to securely access their AWS instances. Gavin Ray, principal engineer at Machinify, explains that this setup caused difficulties: “We were using a VPN solution that was battle-hardened and proven, but it was difficult to scale, operate, and maintain with our small ops team of five engineers at the time.”

Additional downsides of using OpenVPN to access their AWS instances included subjecting Machinify’s architecture to a single point of failure. Gavin and his team also experienced significant cognitive and operational costs making OpenVPN available for every member of their remote-first team — such as handling key management and making sure everything was synchronized across the organization.

Managing user accounts was also a burden, even with automation. HITRUST compliance requirements meant that Machinify had to reset user passwords every 30 days, which often went awry. In many cases, users would forget to reset their passwords and let them expire, leading to downtime and an unnecessary burden on the operations team. Gavin explains: “Once a week, we’d get a ‘Hey, my VPN is broken’ request. So an engineer would have to manually dig through OpenVPN logs (which are designed to be very tightly controlled) and confirm the person was entering the wrong password, and it sucked. We’d have to spend all this time and energy investigating why our users couldn’t access our VPN. It was not pleasant.”

They realized it was time for a change

As Machinify grew and the need for a scalable solution became more important, Gavin and his team decided to reevaluate their VPN. Most importantly, Machinify wanted to minimize time spent on maintenance while still meeting the security standards required to earn HITRUST certification.

Moreover, because Machinify works in the healthcare industry, they are subject to strict regulatory standards as they work toward HITRUST certification — under HIPAA guidelines, for example.

Gavin was already aware of Tailscale, so he tried it out on his personal network. “I travel," says Gavin, “and I need to access stuff on the fly. I tried Tailscale and — oh my God! I could get into my servers from anywhere in the world. It was amazing. It was like click, and it’s done. It just worked.”

The logical next step was to roll out Tailscale at Machinify, but it was a hard sell — at first. “People in the security industry are hesitant to depart from the tried-and-true,” Gavin says. “And even though WireGuard [which Tailscale is built on] is certainly secure, it hasn’t been in the wild long enough to have proven its merits.”

Machinify started with a limited rollout for a small number of engineers, and the tune soon changed. The simplicity and ease of the onboarding process was a big factor. Gavin says, “How easy was it to implement Tailscale? I clicked a button, and it just worked.”

On top of that, their operations team saw a dramatic reduction in complications with their VPN and connectivity. There was also such a conspicuous lack of “Hey, my VPN is broken” requests that they quickly decided to roll Tailscale out to the entire organization.

There was no setup, no configuration, and we had it rolled out across our entire fleet in an hour. Frankly, it was the most joyous experience I’ve had with any commercial product.
Gavin Ray Principal Engineer

Rolling out Tailscalse at Machinify was a win for everyone

Before, Machinify found it cumbersome to manage keys and administer their VPN, even after investing heavily in automation. “When somebody would leave the company, we had to change the automation in multiple places. But with Tailscale, we use Okta group sync: We only have to remove somebody’s access through the control plane. Click — and it’s done.” Having user accounts tied to their identity provider also made it that much simpler to identify who had access to what, and to manage access, with users and groups synced automatically.

In addition to fast and easy onboarding, Machinify especially appreciates how the administrative and operational pain points of their old VPN were abstracted into a centralized and easy-to-use control plane. “Before,” Gavin explains, “if you had access to our VPN, you had the keys to the kingdom. With Tailscale’s control plane, we can finely control scope. That granular control has greatly improved our security and significantly reduced administration costs.”

Combined with using Tailscale’s integrated Okta authentication, these advantages have allowed the number of administrators overseeing their VPN to go from five people to effectively zero, and reduced the time commitment from several hours a week for virtually none. Tailscale’s exit nodes have proven especially beneficial with regard to Machinify’s HITRUST requirements. Gavin explains: “Because HIPAA-type data has to be tightly controlled, we have some instances that go through a single bastion host that we can monitor. So we have a host in AWS that is tagged to allow access to that particular subnet.” Using exit nodes like this allows Gavin’s team to easily restrict access as needed for compliance.

But the coolest feature of Tailscale, for Gavin, is NAT traversal. With his travel schedule, he can access any machine at any time, no matter where he is. What if he misses a flight and is stuck somewhere in Europe? “I don’t care. Tailscale affords me the ability to do anything that I need to do for my job. It lets anybody on my team do the same. It is by far the most useful tool I have, aside from compilers.”

But Tailscale SSH goes above and beyond for a VPN!

As part of their user management strategy, Machinify also uses Tailscale SSH and ACL tags to manage access to their 2,000+ AWS instances. Tailscale SSH lets Machinify grant permission to users who need access to these environments, and gives Gavin the ability to track who did what, which commands they ran, and when. Gavin adds, “The tags that we use for SSH access are essentially this: You have access to our AWS instances, or you don’t.” Because Tailscale’s ACLs are default deny, Machinify explicitly defines the access they want in their network. Gavin points out, “There wasn’t the ability to have routes bleed, because everything had to be tagged as part of the mesh network.”

Tailscale SSH also helps Machinify satisfy their HITRUST certification requirements. “That has been exceptionally helpful,” Gavin says, “especially in the context of HITRUST compliance and audits. We now have a complete audit log of everyone connected to Tailscale — for example, we can show user X went into machine Y and did A, B, and C before disconnecting. Having that auditability has saved us a tremendous amount of time and energy, and has allowed us to meet our compliance requirement.”

That’s why Machinify hearts Tailscale

After making the switch to Tailscale, Machinify hasn’t looked back. “We don’t need more Tailscale features,” says Gavin. “We are paying you simply because of the value you deliver to us on a regular basis. A hosted, old-style VPN solution, even without all the marvelous features Tailscale has, is 10x what you charge. Tailscale is absolutely phenomenal, and the value is exceptional.”