[QUESTION] What would be your strategy for Bastille HA/Cluster/DR on multiple servers ?

[2Belette]

Hi,

I am following all the improvements from the last past months and very happy to see all the changes after a quiet period.

I am evaluating Bastille as I want to get rid of Linux and my servers are now running on FreeBSD and was using Incus which is quite good for managing Linux containers, and also have a good and nice clustering feature.

I am wondering what would be your thoughts on how to create a kind of Bastille cluster ?

For Network, I am thinking of having one jail per server (I have 3x servers) and get BGP for that portion.

Where I am struggling, is on the jails themselves if there is a simple way to duplicate or manage them globally would zfs send an option/viable ?

I would love to listen to your ideas/suggestions, everything around clustering/HA/DR are welcomed, maybe some of you have the same need or already being able to address it!

Many thanks

[tschettervictor]

Since Bastille is just a set of scripts that runs commands around the jail framework, this would require some serious thought.

This has been asked before, and I don't think a viable solution has been given.

There would need to be a central management jail or system that with orchestrate all this, and it would have to know everything about each system you run and all its jails. Sort of like a load balancer...

[bmac2]

I would look at nomad, probably. it clusters jails. Clustering is way above the abilities of our shell scripts. You need the orchestration of something like Nomad. As far as I can find on looking, nomad is one of the only choices.

[2Belette]

Thanks, yes I saw Nomad but when I saw that the latest update was 6 years ago (FreeBSD 12.0..) I don't think this would be a good idea..

Cluster is ideal but not mandatory as long as some automatic task can be done, at least for my usecase

What about automatic backup instead of real cluster, would it be possible to make use of zfs command to send from one server to the second and have Bastille to automatically import it maybe ? (I never played with zfs outside of the same system but it seems possible).

[tschettervictor]

It's definitely possible. I use a small script/pkg called "zfs-replicate" that does my backups. Set it up once then run it as a cron job daily at 12 am.

It's available in pkg.

[2Belette]

I will definitely have a look! Thanks for that.

Do you do anything on Bastille to simplify / automatize the import ?

[tschettervictor]

I will definitely have a look! Thanks for that.

Do you do anything on Bastille to simplify / automatize the import ?

Nope. Just set to back up the whole pool to my backup server.

[tschettervictor]

There is also a tool available called "zrep" which is designed for real time synchronization and failover.

https://bolthole.com

https://github.com/bolthole/zrep

[2Belette]

Thanks, I am discovering it ! I will definitely make a try on my lab

[yaazkal]

Thanks, yes I saw Nomad but when I saw that the latest update was 6 years ago (FreeBSD 12.0..) I don't think this would be a good idea..

Could you clarify what you mean by that? The Nomad package in FreeBSD is regularly updated. The current version is 1.9.3.

[2Belette]

Indeed when I search I took a look on the "official jail task driver for FreeBSD" from the Nomad website : https://github.com/cneira/jail-task-driver mentioned here : https://developer.hashicorp.com/nomad/plugins/drivers/community/jail-task-driver

So does the Nomad in ports which is updated is the server side and it does have an updated version of the jail driver ? (Sorry I am new to Nomad and have to take a look to understand the architecture)

[yaazkal]

Oh I now understand. That driver is for jails directly. Nomad will need a BastilleBSD driver that still does not exist. I know that pot has a nomad driver: https://github.com/bsdpot/nomad-pot-driver

[2Belette]

That would be nice indeed to have a driver for BastilleBSD !
I am wondering how nomad with pot is able to have access to the host resource to properly load-balance/load-share ...

Perhaps a first good step would be for BastilleBSD to be compatible with Terraform/OpenTofu so jail creation with Bastille would be programmatically possible

[yaazkal]

Perhaps a first good step would be for BastilleBSD to be compatible with Terraform/OpenTofu so jail creation with Bastille would be programmatically possible

For that you can try ansible with this role for Bastille: https://github.com/yaazkal/ansible-role-bastille

[2Belette]

Thanks ! I see your repo is 2 years old and 13.2 is mentioned, all is working fine with current 14.2 and latest version of Bastille? (I am thinking of all the recents good change and the incoming ones with multiples interfaces options for example)

[yaazkal]

Thanks ! I see your repo is 2 years old and 13.2 is mentioned, all is working fine with current 14.2 and latest version of Bastille? (I am thinking of all the recents good change and the incoming ones with multiples interfaces options for example)

It should work for the basic tasks, but I can spare some time testing in the next weeks (for the new features)

[tschettervictor]

https://developer.hashicorp.com/nomad/docs/concepts/plugins

If ever someone wants to write a nomad driver for Bastille.