Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update core #301

Merged
merged 18 commits into from
Jun 9, 2021
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -3,3 +3,8 @@ up
hosts/up-*
.direnv
doc/index.html

# Result of flk commands
vm
iso
doi
5 changes: 4 additions & 1 deletion doc/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Pull Requests
If making a change to core, or adding a feature, please be sure to update the
All development is done in the `develop` branch. Only minor bug-fixes and release
PRs should target `master`.

If making a change to the template, or adding a feature, please be sure to update the
relevant docs. Each directory contains its own README.md, which will
automatically be pulled into the [mdbook](https://devos.divnix.com). The book is
rendered on every change, so the docs should always be up to date.
1 change: 1 addition & 0 deletions doc/SUMMARY.md
Original file line number Diff line number Diff line change
@@ -24,6 +24,7 @@
- [up](./flk/up.md)
- [update](./flk/update.md)
- [get](./flk/get.md)
- [doi](./flk/doi.md)
- [iso](./flk/iso.md)
- [install](./flk/install.md)
- [home](./flk/home.md)
11 changes: 6 additions & 5 deletions doc/concepts/hosts.md
Original file line number Diff line number Diff line change
@@ -6,13 +6,14 @@ of these hosts, devos automatically imports every _.nix_ file inside this
directory to the mentioned attribute set, applying the projects defaults to
each. The only hard requirement is that the file contain a valid NixOS module.

As an example, a file `hosts/system.nix` will be available via the flake
output `nixosConfigurations.system`. You can have as many hosts as you want
and all of them will be automatically imported based on their name.
As an example, a file `hosts/system.nix` or `hosts/system/default.nix` will
be available via the flake output `nixosConfigurations.system`. You can have
as many hosts as you want and all of them will be automatically imported based
on their name.

For each host, the configuration automatically sets the `networking.hostName`
attribute to the name of the file minus the _.nix_ extension. This is for
convenience, since `nixos-rebuild` automatically searches for a configuration
attribute to the folder name or name of the file minus the _.nix_ extension. This
is for convenience, since `nixos-rebuild` automatically searches for a configuration
matching the current systems hostname if one is not specified explicitly.

You can set channels, systems, and add extra modules to each host by editing the
7 changes: 4 additions & 3 deletions doc/concepts/overrides.md
Original file line number Diff line number Diff line change
@@ -27,10 +27,11 @@ You can also pull modules from other channels. All modules have access to the
`modulesPath` for each channel as `<channelName>ModulesPath`. And you can use
`disabledModules` to remove modules from the current channel.

Pulling the zsh module from the `latest` channel:
To pull zsh module from the `latest` channel this code can be placed in any module, whether its your host file, a profile, or a module in ./modules etc:
```nix
{ latestModulesPath }: {
modules = [ "${latestModulesPath}/programs/zsh/zsh.nix" ];
{ latestModulesPath }:
{
imports = [ "${latestModulesPath}/programs/zsh/zsh.nix" ];
disabledModules = [ "programs/zsh/zsh.nix" ];
}
```
43 changes: 24 additions & 19 deletions doc/concepts/profiles.md
Original file line number Diff line number Diff line change
@@ -8,42 +8,37 @@ separation of concerns.
If you need guidance, a community [branch](https://github.com/divnix/devos/tree/community/profiles)
is maintained to help get up to speed on their usage.

## Constraints
For the sake of consistency, a profile should always be defined in a
___default.nix___ containing a [nixos module config][config].
A profile's directory is used for quick modularization of
[interelated bits](./profiles.md#subprofiles).
## Creation
Profiles are created with the `rakeLeaves` function which recursively collects
`.nix` files from within a folder. The recursion stops at folders with a `default.nix`
in them. You end up with an attribute set with leaves(paths to profiles) or
nodes(attrsets leading to more nodes or leaves).

A profile is used for quick modularization of [interelated bits](./profiles.md#subprofiles).

> ##### _Notes:_
> * For _declaring_ module options, there's the [modules](../outputs/modules.md) directory.
> * This directory takes inspiration from
> [upstream](https://github.com/NixOS/nixpkgs/tree/master/nixos/modules/profiles)
> .
> * Sticking to a simple [spec][spec] has refreshing advantages.
> [hercules-ci](../integrations/hercules.md) expects all profiles to be
> defined in a ___default.nix___, allowing them to be built automatically when
> added. Congruently, [suites](suites.md) expect ___default.nix___ to avoid
> having to manage their paths manually.
## Subprofiles
Profiles can also define subprofiles. They follow the same constraints outlined
above. A good top level profile should be a high level concern, such as your
personal development environment while the subprofiles should be more focused
program configurations such as your text editor, and shell configs. This way,
you can either pull in the whole development profile, or pick and choose
individual programs.
### Nested profiles
Profiles can be nested in attribute sets due to the recursive nature of `rakeLeaves`.
This can be useful to have a set of profiles created for a specific purpose. It is
sometimes useful to have a `common` profile that has high level concerns related
to all its sister profiles.

### Example

profiles/develop/default.nix:
profiles/develop/common.nix:
```nix
{
imports = [ ./zsh ];
# some generic development concerns ...
}
```

profiles/develop/zsh/default.nix:
profiles/develop/zsh.nix:
```nix
{ ... }:
{
@@ -52,6 +47,16 @@ profiles/develop/zsh/default.nix:
}
```

The examples above will end up with a profiles set like this:
```nix
{
develop = {
common = ./profiles/develop/common.nix;
zsh = ./profiles/develop/zsh.nix;
};
}
```

## Conclusion
Profiles are the most important concept in DevOS. They allow us to keep our
Nix expressions self contained and modular. This way we can maximize reuse
17 changes: 5 additions & 12 deletions doc/concepts/suites.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,11 @@
# Suites
Suites provide a mechanism for users to easily combine and name collecitons of
profiles. For good examples, check out the suites defined in the community
[branch](https://github.com/divnix/devos/blob/community/suites/default.nix).
profiles. For good examples, check out the suites defined in the community branch.

In the future, we will use suites as a mechanism for deploying various machine
types which don't depend on hardware, such as vm's and containers.

They are defined with the `suites` argument in either `home` or `nixos` namespace.
Suites should be passed as a function that take profiles as an argument.

The profiles are passed based on the folder names and list passed to the relevant
`profiles` argument. In the template's flake.nix `profiles` is set as
`[ ./profiles ./users ]` and that corresponds to the `{ profiles, users }` argument
pattern.
`suites` are defined in the `importables` argument in either the `home` or `nixos`
namespace. They are a special case of an `importable` which get passed as a special
argument (one that can be use in an `imports` line) to your hosts. All lists defined
in `suites` are flattened and type-checked as paths.

## Definition
```nix
23 changes: 17 additions & 6 deletions doc/concepts/users.md
Original file line number Diff line number Diff line change
@@ -23,11 +23,23 @@ your users. For a fully fleshed out example, check out the developers personal
```

## Home Manager
Home Manager support follows the same principles as regular nixos configurations.
Home Manager support follows the same principles as regular nixos configurations,
it even gets its own namespace in your `flake.nix` as `home`.

All modules defined in [user modules][modules-list] will be imported to
Home Manager. All profiles are availabe in [suites][suites] as userProfiles.
The `userSuites` output will be available in your Home Manager Configuration as
the special argument, `suites`.
Home Manager.
User profiles can be collected in a similar fashion as system ones into a `suites`
argument that gets passed to your home-manager users.

### Example
```nix
{
home-manager.users.nixos = { suites, ... }: {
imports = suites.base;
};
}
```


## External Usage
You can easily use the defined home-manager configurations outside of NixOS
@@ -56,5 +68,4 @@ nix build "github:divnix/devos#[email protected]
```

[home-manager]: https://nix-community.github.io/home-manager
[suites]: https://github.com/divnix/devos/tree/core/suites/default.nix
[modules-list]: https://github.com/divnix/devos/tree/core/modules/module-list.nix
[modules-list]: https://github.com/divnix/devos/tree/core/users/modules/module-list.nix
10 changes: 10 additions & 0 deletions doc/flk/doi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# DigitalOcean

Now you can [create a droplet](https://cloud.digitalocean.com/droplets/new) using your custom image.

Making a DigitalOcean compatible image for `hosts/NixOS.nix` is as simple as:
```sh
flk doi NixOS
```

This works for any file matching `hosts/*.nix` excluding `default.nix`.
87 changes: 0 additions & 87 deletions doc/lib.md

This file was deleted.

192 changes: 163 additions & 29 deletions doc/mkFlakeOptions.md
Original file line number Diff line number Diff line change
@@ -40,7 +40,7 @@ nix flake

*_Default_*
```
"inputs.<name>"
"self.inputs.<name>"
```


@@ -81,6 +81,56 @@ attribute set or path convertible to it



## devshell
Modules to include in your devos shell. the `modules` argument
will be exported under the `devshellModules` output


*_Type_*:
submodule


*_Default_*
```
{}
```




## devshell.externalModules
modules to include that won't be exported
meant importing modules from external flakes


*_Type_*:
list of valid module or path convertible to its or anything convertible to it


*_Default_*
```
[]
```




## devshell.modules
modules to include in all hosts and export to devshellModules output


*_Type_*:
list of path to a modules or anything convertible to it or path convertible to it


*_Default_*
```
[]
```




## home
hosts, modules, suites, and profiles for home-manager

@@ -103,7 +153,7 @@ meant importing modules from external flakes


*_Type_*:
list of valid module or path convertible to its
list of valid module or path convertible to its or anything convertible to it


*_Default_*
@@ -114,31 +164,40 @@ list of valid module or path convertible to its



## home.modules
modules to include in all hosts and export to homeModules output
## home.importables
Packages of paths to be passed to modules as `specialArgs`.


*_Type_*:
list of path to a modules or anything convertible to it or path convertible to it
attribute set


*_Default_*
```
[]
{}
```




## home.profiles
profile folders that can be collected into suites
the name of the argument passed to suites is based
on the folder name.
[ ./profiles ] => { profiles }:
## home.importables.suites
collections of profiles


*_Type_*:
list of paths
attribute set of list of paths or anything convertible to its






## home.modules
modules to include in all hosts and export to homeModules output


*_Type_*:
list of path to a modules or anything convertible to it or path convertible to it


*_Default_*
@@ -149,30 +208,48 @@ list of paths



## home.suites
Function that takes profiles and returns suites for this config system
These can be accessed through the 'suites' special argument.
## home.profiles
WARNING: The 'suites' and `profiles` options have been deprecated, you can now create
both with the importables option. `rakeLeaves` can be used to create profiles and
by passing a module or `rec` set to `importables`, suites can access profiles.
Example:
```
importables = rec {
profiles = digga.lib.importers.rakeLeaves ./profiles;
suites = with profiles; { };
}
```
See https://github.com/divnix/digga/pull/30 for more details


*_Type_*:
function that evaluates to a(n) attrs or path convertible to it
list of paths


*_Default_*
```
"<function>"
[]
```




## inputs
inputs for this flake
used to set channel defaults and create registry
## home.suites
WARNING: The 'suites' and `profiles` options have been deprecated, you can now create
both with the importables option. `rakeLeaves` can be used to create profiles and
by passing a module or `rec` set to `importables`, suites can access profiles.
Example:
```
importables = rec {
profiles = digga.lib.importers.rakeLeaves ./profiles;
suites = with profiles; { };
}
```
See https://github.com/divnix/digga/pull/30 for more details


*_Type_*:
attribute set of nix flakes
function that evaluates to a(n) attrs or path convertible to it



@@ -236,7 +313,7 @@ meant importing modules from external flakes


*_Type_*:
list of valid module or path convertible to its
list of valid module or path convertible to its or anything convertible to it


*_Default_*
@@ -343,11 +420,46 @@ null



## nixos.importables
Packages of paths to be passed to modules as `specialArgs`.


*_Type_*:
attribute set


*_Default_*
```
{}
```




## nixos.importables.suites
collections of profiles


*_Type_*:
attribute set of list of paths or anything convertible to its






## nixos.profiles
profile folders that can be collected into suites
the name of the argument passed to suites is based
on the folder name.
[ ./profiles ] => { profiles }:
WARNING: The 'suites' and `profiles` options have been deprecated, you can now create
both with the importables option. `rakeLeaves` can be used to create profiles and
by passing a module or `rec` set to `importables`, suites can access profiles.
Example:
```
importables = rec {
profiles = digga.lib.importers.rakeLeaves ./profiles;
suites = with profiles; { };
}
```
See https://github.com/divnix/digga/pull/30 for more details


*_Type_*:
@@ -363,17 +475,39 @@ list of paths


## nixos.suites
Function that takes profiles and returns suites for this config system
These can be accessed through the 'suites' special argument.
WARNING: The 'suites' and `profiles` options have been deprecated, you can now create
both with the importables option. `rakeLeaves` can be used to create profiles and
by passing a module or `rec` set to `importables`, suites can access profiles.
Example:
```
importables = rec {
profiles = digga.lib.importers.rakeLeaves ./profiles;
suites = with profiles; { };
}
```
See https://github.com/divnix/digga/pull/30 for more details


*_Type_*:
function that evaluates to a(n) attrs or path convertible to it






## outputsBuilder
builder for flake system-spaced outputs
The builder gets passed an attrset of all channels


*_Type_*:
function that evaluates to a(n) attrs


*_Default_*
```
"<function>"
"channels: { }"
```


3 changes: 1 addition & 2 deletions doc/outputs/pkgs.md
Original file line number Diff line number Diff line change
@@ -7,8 +7,7 @@ The only minor difference is that, instead of adding the `callPackage` call to
`all-packages.nix`, you just add it the the _default.nix_ in this directory,
which is defined as a simple overlay.

This overlay is set as the default `overlay` output attribute for the flake.
And all the packages are exported via `packages.<system>.<pkg-name>`, for all
All the packages are exported via `packages.<system>.<pkg-name>`, for all
the supported systems listed in the package's `meta.platforms` attribute.

And, as usual, every package in the overlay is also available to any NixOS
110 changes: 101 additions & 9 deletions doc/secrets.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,110 @@
# Secrets
Secrets are managed using [git-crypt][git-crypt] so you can keep your flake in
a public repository like GitHub without exposing your password or other
sensitive data.
Secrets are managed using [git-crypt][git-crypt] and [agenix][agenix]
so you can keep your flake in a public repository like GitHub without
exposing your password or other sensitive data.

By default, everything in the secrets folder is automatically encrypted. Just
be sure to run `git-crypt init` before putting anything in here.

## Agenix
Currently, there is [no mechanism][secrets-issue] in nix itself to deploy secrets
within the nix store because it is world-readable.

Most NixOS modules have the ability to set options to files in the system, outside
the nix store, that contain sensitive information. You can use [agenix][agenix]
to easily setup those secret files declaratively.

[agenix][agenix] encrypts secrets and stores them as .age files in your repository.
Age files are encrypted with multiple ssh public keys, so any host or user with a
matching ssh private key can read the data. The [age module][age module] will add those
encrypted files to the nix store and decrypt them on activation to `/run/secrets`.

### Setup
All hosts must have openssh enabled, this is done by default in the core profile.

You need to populate your `secrets/secrets.nix` with the proper ssh public keys.
Be extra careful to make sure you only add public keys, you should never share a
private key!!

secrets/secrets.nix:
```nix
let
system = "<system ssh key>";
user = "<user ssh key>";
allKeys = [ system user ];
in
```

On most systems, you can get your systems ssh public key from `/etc/ssh/ssh_host_ed25519_key.pub`. If
this file doesn't exist you likely need to enable openssh and rebuild your system.

Your users ssh public key is probably stored in `~/.ssh/id_ed25519.pub` or
`~/.ssh/id_rsa.pub`. If you haven't generated a ssh key yet, be sure do so:
```sh
ssh-keygen -t ed25519
```

> ##### _Note:_
> The underlying tool used by agenix, rage, doesn't work well with password protected
> ssh keys. So if you have lots of secrets you might have to type in your password many
> times.

### Secrets
You will need the `agenix` command to create secrets. DevOS conveniently provides that
in the devShell, so just run `nix develop` whenever you want to edit secrets. Make sure
to always run `agenix` while in the `secrets/` folder, so it can pick up your `secrets.nix`.

To create secrets, simply add lines to your `secrets/secrets.nix`:
```
let
...
allKeys = [ system user ];
in
{
"secret.age".publicKeys = allKeys;
}
```
That would tell agenix to create a `secret.age` file that is encrypted with the `system`
and `user` ssh public key.

Then go into the `secrets` folder and run:
```sh
agenix -e secret.age
```
This will create the `secret.age`, if it doesn't already exist, and allow you to edit it.

If you ever change the `publicKeys` entry of any secret make sure to rekey the secrets:
```sh
agenix --rekey
```

### Usage
Once you have your secret file encrypted and ready to use, you can utilize the [age module][age module]
to ensure that your secrets end up in `/run/secrets`.

In any profile that uses a NixOS module that requires a secret you can enable a particular secret like so:

```nix
{ self, ... }:
{
age.secrets.mysecret.file = "${self}/secrets/mysecret.age";
}
```


Then you can just pass the path `/run/secrets/mysecret` to the module.

You can make use of the many options provided by the age module to customize where and how
secrets get decrypted. You can learn about them by looking at the
[age module][age module].


> ##### _Note:_
> Currently, there is [no mechanism][secrets-issue] in nix to deploy secrets
> within the nix/store so, if they end up in the nix/store after deployment, they
> will be world readable on that machine.
>
> The author of devos intends to implement a workaround for this situation in
> the near future, but for the time being, simple be aware of this.
> You can take a look at the [agenix repository][agenix] for more information
> about the tool.
[git-crypt]: https://github.com/AGWA/git-crypt
[agenix]: https://github.com/ryantm/agenix
[age module]: https://github.com/ryantm/agenix/blob/master/modules/age.nix
[secrets-issue]: https://github.com/NixOS/nix/issues/8
11 changes: 4 additions & 7 deletions doc/tests.md
Original file line number Diff line number Diff line change
@@ -5,22 +5,19 @@ NixOS offers some incredibly powerful tools to write tests for your
configuration, and, optionally, run them in
[CI](./integrations/hercules.md).

## Lib Tests
You can easily write tests for your own library functions in the
lib/___tests/lib.nix___ file and they will be run on every `nix flake check` or
during a CI run.

## Unit Tests
Unit tests are can be created from regular derivations, and they can do
Unit tests can be created from regular derivations, and they can do
almost anything you can imagine. By convention, it is best to test your
packages during their [check phase][check]. All packages and their tests will
be built during CI.

## Integration Tests
All your profiles defined in suites will be tested in a NixOS VM.

You can write integration tests for one or more NixOS VMs that can,
optionally, be networked together, and yes, it's as awesome as it sounds!

Be sure to use the `mkTest` function, in the [___tests/default.nix___][default]
Be sure to use the `mkTest` function from digga, `digga.lib.pkgs-lib.mkTest`
which wraps the official [testing-python][testing-python] function to ensure
that the system is setup exactly as it is for a bare DevOS system. There are
already great resources for learning how to use these tests effectively,
40 changes: 31 additions & 9 deletions flake.lock
41 changes: 33 additions & 8 deletions flake.nix
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@
{
nixos.url = "nixpkgs/nixos-unstable";
latest.url = "nixpkgs";
digga.url = "github:divnix/digga";
digga.url = "github:divnix/digga/master";

ci-agent = {
url = "github:hercules-ci/hercules-ci-agent";
@@ -17,13 +17,26 @@
home.inputs.nixpkgs.follows = "nixos";
naersk.url = "github:nmattia/naersk";
naersk.inputs.nixpkgs.follows = "latest";
agenix.url = "github:ryantm/agenix";
agenix.inputs.nixpkgs.follows = "latest";
nixos-hardware.url = "github:nixos/nixos-hardware";

pkgs.url = "path:./pkgs";
pkgs.inputs.nixpkgs.follows = "nixos";
};

outputs = inputs@{ self, pkgs, digga, nixos, ci-agent, home, nixos-hardware, nur, ... }:
outputs =
{ self
, pkgs
, digga
, nixos
, ci-agent
, home
, nixos-hardware
, nur
, agenix
, ...
} @ inputs:
digga.lib.mkFlake {
inherit self inputs;

@@ -36,6 +49,7 @@
./pkgs/default.nix
pkgs.overlay # for `srcs`
nur.overlay
agenix.overlay
];
};
latest = { };
@@ -60,6 +74,7 @@
{ _module.args.ourLib = self.lib; }
ci-agent.nixosModules.agent-profile
home.nixosModules.home-manager
agenix.nixosModules.age
./modules/customBuilds.nix
];
};
@@ -69,21 +84,31 @@
/* set host specific properties here */
NixOS = { };
};
profiles = [ ./profiles ./users ];
suites = { profiles, users, ... }: with profiles; rec {
base = [ core users.nixos users.root ];
importables = rec {
profiles = digga.lib.importers.rakeLeaves ./profiles // {
users = digga.lib.importers.rakeLeaves ./users;
};
suites = with profiles; rec {
base = [ core users.nixos users.root ];
};
};
};

home = {
modules = ./users/modules/module-list.nix;
externalModules = [ ];
profiles = [ ./users/profiles ];
suites = { profiles, ... }: with profiles; rec {
base = [ direnv git ];
importables = rec {
profiles = digga.lib.importers.rakeLeaves ./users/profiles;
suites = with profiles; rec {
base = [ direnv git ];
};
};
};

devshell.externalModules = { pkgs, ... }: {
packages = [ pkgs.agenix ];
};

homeConfigurations = digga.lib.mkHomeConfigurations self.nixosConfigurations;

deploy.nodes = digga.lib.mkDeployNodes self.nixosConfigurations { };
5 changes: 4 additions & 1 deletion modules/customBuilds.nix
Original file line number Diff line number Diff line change
@@ -10,7 +10,6 @@ in
system.build = {
iso = (mkBuild (diggaLib.modules.isoConfig {
inherit self;
inherit (self) inputs;
fullHostConfig = config;
})).config.system.build.isoImage;

@@ -26,5 +25,9 @@ in
}
];
})).config.home-manager.users;

digitalOcean = (mkBuild ({ modulesPath, ... }: {
imports = [ "${modulesPath}/virtualisation/digital-ocean-image.nix" ];
})).config.system.build.digitalOceanImage;
};
}
1 change: 1 addition & 0 deletions overlays/overrides.nix
Original file line number Diff line number Diff line change
@@ -8,6 +8,7 @@ channels: final: prev: {
discord
element-desktop
manix
rage
nixpkgs-fmt
qutebrowser
signal-desktop
6 changes: 6 additions & 0 deletions profiles/core/default.nix
Original file line number Diff line number Diff line change
@@ -143,6 +143,12 @@ in
'';
};

# For rage encryption, all hosts need a ssh key pair
services.openssh = {
enable = true;
openFirewall = lib.mkDefault false;
};

services.earlyoom.enable = true;

}
1 change: 1 addition & 0 deletions secrets/.gitattributes
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
* filter=git-crypt diff=git-crypt
.gitattributes !filter !diff
secrets.nix !filter !diff
README.md !filter !diff
9 changes: 9 additions & 0 deletions secrets/secrets.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
let
# set ssh public keys here for your system and user
system = "";
user = "";
allKeys = [ system user ];
in
{
"secret.age".publicKeys = allKeys;
}