Discussion: using package.el and straight.el together

[rgrinberg]

straight.el looks like a real awesome package manager and I'd like to start using it right away. But I already have a massive spacemacs based config, so I'm wondering if I can make a gradual change towards straight somehow. For example, only using it to manage a few packages to start. Wondering if this is a good idea and is possible at all.

[raxod502]

Well, I would say that straight.el is much better than package.el in this respect, since straight.el will do only what you explicitly tell it to do in your init-file, and nothing more. As opposed to package.el which insists on activating all the packages you ever installed with it.

There should be no trouble at all with using both straight.el and other package managers, at least from the straight.el side. If you are going to be using straight.el together with package.el, you may want to set straight-enable-package-integration to nil, since otherwise straight.el will automatically disable some "features" (i.e. bugs which are defended vehemently on the mailing list) of package.el that cause package.el to modify the init-file.

The thing you have to be concerned about is duplicate packages. There is no inherent harm in loading a package twice, other than that your init may be slower and you might get some weird side effects if the two versions being loaded are not the same. You should be especially concerned about dependencies, since if you use package manager X to load package Y, then you also need to use package manager X to load all of the dependencies of package Y—otherwise you'll get duplicate packages if you load those dependencies also using package manager Z.

Another consideration is that use-package does not have support for using different package managers for the :ensure clauses in different use-package forms. This is not a problem straight.el can fix, although you may want to keep an eye on my new with-feature package, which will solve this problem among others. As a workaround, I would recommend disabling :ensure for the use-package forms that you wish to use straight.el, and then just putting a manual (straight-use-package 'whatever) above those use-package forms. I.e.

(straight-use-package 'el-patch)
(use-package el-patch
  :ensure nil
  <whatever>)

instead of

(use-package el-patch
  :ensure t
  <whatever>

In order for the use-package forms that do specify :ensure to keep using package.el rather than straight.el, you should set straight-enable-use-package-integration to nil.

Does this answer your question? Let me know if any other information would be helpful.

[raxod502]

Also, for the question of "is it safe to use" in general, see the new known issue FAQ section.

[caadar]

Hi, what is "different use-package forms" you mentioned?

Are you recommend to use such style

(straight-use-package 'el-patch)
(use-package el-patch
 :ensure nil
 <whatever>)

for every package installed?

Currently I set config with

(use-package el-patch
 :ensure t
 <whatever>)

only. It seems works.

[vyp]

@caadar No, that's just for @rgrinberg's particular use case. What you're doing is already recommended. 😀

[raxod502]

that's just for @rgrinberg's particular use case

Correct, the problem only arises when you want to use straight.el for some packages and another package manager for others.

[rgrinberg]

@raxod502 thanks for the reassuring response. I'm trying out straight.el and things are going great except for one situation. There are some packages where I'd like straight to simply install them from an existing local repositories instead of trying to clone them. Those are usually packages that aren't just emacs packages and hence need to be located where they are in other (non emacs) contexts. For example, the utop package that I'm trying to get working:

(straight-use-package
 '(utop
   :type git :repo "rgrinberg/utop"
   :local-repo "/home/rgrinberg/reps/ml/utop"
   :files "src/top/utop.el"
   :upstream (:host github :repo "diml/utop")))

Ideally, straight-el should simply assume that this package is cloned and just build it. But I can't figure out how to do that.

[raxod502]

Currently straight.el only supports local repositories that are inside ~/.emacs.d/straight/repos. That is, :local-repo isn't resolved as a real path, it's just concatenated to the prefix I mentioned. So that's your problem.

The quick fix is to symlink the repository into ~/.emacs.d/straight/repos. In general, I would consider allowing a :local-repo that is outside of your ~/.emacs.d directory, if that would be a useful feature.

[rgrinberg]

Thanks, this workaround is adequate. I think straight supporting local repos anywhere on the filesystem would be a useful feature.

[raxod502]

I've created #133 to track this feature separately. Feel free to either close this issue or follow up with other questions.

[rgrinberg]

Thanks again :) I will create new issues/questions as necessary

[rgrinberg]

Actually, I have another question that probably doesn't deserve its own issue:

Is there a way to prevent straight from hacking use-package? The reason for that is that I already have use-package calls done by my spacemacs config and those packages are already being installed by the traditional package management system. It would be preferable would only install packages that are used through straight-use-package only.

[raxod502]

(setq straight-enable-use-package-integration nil)

for more info, Integration with use-package. That's what I advise in #128 (comment), to allow for the default to remain package.el but for some packages to be installed via straight.el.

[rgrinberg]

OK, thanks.

[raxod502]

… except for the part where straight-enable-use-package-integration doesn't actually do anything(?!!?!), not sure how I screwed that up.

[raxod502]

Fixed on develop.

[emacs18]

@raxod502 said "package.el which insists on activating all the packages you ever installed with it."

I would like to point out that this is not necessarily the case if you call package-initialize with non-nil optional argument, e.g.,

(package-initialize 'no-activate)

This is how I have always use package.el so that I can have many installed packages, but activate only some of them by using a different init file when emacs starts.

[raxod502]

@emacs18 You're right. It's only the default behavior which is flawed, at least in this case.

If you have a suggestion for a corresponding update to the comparisons section in the README, I'd be glad to consider it.

[emacs18]

@raxod502 I'll look into providing a patch later.

In the mean time, I would like to ask you if the following could allow me to use straight.el along with spacemacs without duplicating packages:

(defun my--advice-package-installed-p (original-function &rest args)
  "Return t if package is installed via `straight' package manager."
  (or (and (fboundp 'straight--dir)
           (straight--dir "build" (symbol-name (car args))))
      (apply original-function args)))

(advice-add 'package-installed-p :around 'my--advice-package-installed-p)

I'm in the process of testing this idea. I also considered writing use-package-ensure-straight similar to use-package-ensure-elpa to be set as the value of use-package-ensure-function variable. However just advising package-installed-p seemed like a simpler idea.

[raxod502]

@emacs18 Your advice doesn't take into account the case where a package was previously loaded using straight.el, but is not currently. Use this instead:

(defun my--advice-package-installed-p (original-function &rest args)
  "Return t if package is installed via `straight' package manager."
  (or (gethash straight--recipe-cache (car args))
      (apply original-function args)))

(advice-add 'package-installed-p :around 'my--advice-package-installed-p)

At some point it may be a good idea to expose some user-facing functions for getting information from the caches.

That said I can't guarantee that advising package-installed-p will actually stop package.el from loading duplicate packages. In fact, if I had to guess, I would say that duplicate loading will still take place.

By the way, straight.el already provides straight-use-package-ensure-function which it sets as the value of use-package-ensure-function. You would of course have to override this with your own version that delegates to both straight.el and package.el conditionally.

[raxod502]

Actually, your original advice doesn't work at all. It always returns t.

[emacs18]

Yes I forgot to add (file-directory-p) call.
Thanks for your advice. I'll poke around some more.

[rgrinberg]

The way :upstream works doesn't seem to work for a use case that I have. My standard practice for naming remotes is as follows:

• origin - upstream, main repository
• rgrinberg - my own fork

Where I only pull from origin and only push to rgrinberg. Is there a way to make straight respect this convention? Since I do have some emacs package repositories that live outside of straights control.

[raxod502]

;; FIXME: Radon said to use this gethash call rather than straight--dir.
;; However this fails with "Wrong type argument: hash-table-p,
;; org-plus-contrib". So use straight--dir for now.

Lol that's just because I swapped the arguments to gethash. Use

(gethash (car args) straight--recipe-cache)

instead.

;; 'dash' package is cloned under 'dash.el' directory. That is why we
;; check for raw name as well as with ".el" appended.

This shouldn't be necessary if you are checking the build directory, which uses package names rather than local repository names. You would need this if you were checking the repos directory instead, although in that case the correct approach would be to check the :local-repo attribute on the recipe in straight--recipe-cache. In any case this is superfluous since you can just use the fixed gethash command above.

(load "/u/me/Public/straight.el/straight")

I am really confused about why you are doing all this manually instead of using the bootstrap snippet. The latter should have the same effect.

All this said, even though straight.el has significant performance problems (see #9, #118, #119, #129), I only expect loading 320 packages to take 5.8s (extrapolating linearly from my own config). 15 minutes is just ridiculous; there must be something terribly wrong. Let me see if I can reproduce the behavior you describe using your code.

[raxod502]

OK. With the following minified init.el:

(setq package-enable-at-startup nil)
(load (concat user-emacs-directory "straight/bootstrap.el") nil 'nomessage)

(mapcar
 #'straight-use-package
 '(
   ac-ispell ace-jump-helm-line ace-link ace-window adaptive-wrap ag
   aggressive-indent alert anaconda-mode anaphora anzu apiwrap async auctex
   auto-compile auto-complete auto-dictionary auto-highlight-symbol
   auto-yasnippet avy bbdb beacon beacon biblio biblio-core bind-key bind-map
   browse-at-remote clang-format clean-aindent-mode clojure-mode closql
   cmake-mode coffee-mode column-enforce-mode command-log-mode company company
   company-anaconda company-anaconda company-auctex company-c-headers
   company-c-headers company-math company-math company-math company-quickhelp
   company-quickhelp company-shell company-shell company-statistics
   company-statistics company-tern company-tern company-web company-web counsel
   counsel-dash counsel-projectile crux cython-mode dash dash-functional
   default-text-scale deferred define-word demangle-mode diff-hl diminish
   disable-mouse disaster dumb-jump edit-server editorconfig ein elfeed
   elisp-slime-nav elpa-mirror elpy emacsql emacsql-sqlite emmet-mode emms
   engine-mode epkg epl esh-help eshell-prompt-extras eshell-z eval-sexp-fu evil
   evil-anzu evil-args evil-ediff evil-escape evil-exchange evil-iedit-state
   evil-indent-plus evil-lion evil-lisp-state evil-magit evil-matchit evil-mc
   evil-nerd-commenter evil-numbers evil-org evil-search-highlight-persist
   evil-surround evil-tutor evil-visual-mark-mode evil-visualstar
   exec-path-from-shell expand-region eyebrowse f fancy-battery
   fill-column-indicator finalize find-file-in-project fish-mode floobits flx
   flx-ido flycheck flycheck-bashate flycheck-pos-tip flyspell-correct
   flyspell-correct-ivy fringe-helper fuzzy geiser ggtags gh-md ghub ghub+
   git-commit git-gutter git-gutter+ git-gutter-fringe git-gutter-fringe+
   git-link git-messenger git-timemachine gitattributes-mode gitconfig-mode
   gitignore-mode gntp gnuplot golden-ratio google-c-style google-translate
   goto-chg graphviz-dot-mode haml-mode helm helm-ag helm-bibtex
   helm-c-yasnippet helm-company helm-core helm-css-scss helm-dash
   helm-descbinds helm-dictionary helm-flx helm-gitignore helm-gtags helm-make
   helm-pydoc help-fns+ hide-comnt highlight highlight-indentation
   highlight-numbers highlight-parentheses highlight-thing history hl-todo ht
   htmlize hungry-delete hy-mode hydra ibuffer-projectile iedit igrep imenu-list
   impatient-mode indent-guide info+ insert-shebang irony itail ivy ivy-hydra
   ivy-purpose js-doc js2-mode js2-refactor json-mode json-reformat
   json-snatcher key-chord less-css-mode link-hint linum-relative lispy
   live-py-mode livid-mode load-relative loc-changes log4e macrostep magit
   magit-gitflow magit-popup magithub markdown-mode markdown-toc
   math-symbol-lists mmm-mode mmt monokai-theme move-text multi-term
   multiple-cursors neotree ob-ipython open-junk-file org org-alert org-brain
   org-bullets org-download org-mime org-password-manager org-pomodoro
   org-present org-projectile org-tree-slide ox-gfm ox-pandoc ox-reveal p4
   package-build packed pacmacs paradox paredit parent-mode parsebib
   password-generator password-store pcmpl-git pcre2el persp-mode
   pip-requirements pkg-info popup popwin pos-tip powerline projectile pug-mode
   py-isort pyenv-mode pytest pythonic pyvenv quelpa rainbow-delimiters realgud
   request request-deferred restart-emacs s sass-mode scss-mode shell-pop showkey
   simple-call-tree simple-httpd skewer-mode slim-mode smartparens smeargle smex
   spaceline spinner string-inflection sudoku swiper symon tagedit tern
   test-simple tldr toc-org typit undo-tree use-package uuidgen vi-tilde-fringe
   volatile-highlights web-beautify web-completion-data web-mode websocket wgrep
   wgrep-ag which-key window-purpose winum with-editor worf ws-butler xah-lookup
   xterm-color yapfify yasnippet zeal-at-point zoutline
   ))

I observe an M-x emacs-init-time of 3.6 seconds. Can you test with this version @emacs18?

[raxod502]

Note, by the way: you will observe very very poor performance of straight.el in the case that you evaluate a lot of straight-use-package calls without wrapping said evaluation in a transaction. By default, init is considered a transaction. You are encouraged to also insert straight-transaction blocks into any of your interactive commands that evaluate Elisp code interactively.

In particular, your profiling

(elp-instrument-package "straight")
(elp-reset-all)
(mapcar #'straight-use-package '( ... )) ;; about 170 pacakges
(elp-results)

will have terrible performance compared to if you wrap the mapcar in a straight-transaction. This is due to several reasons, but the biggest is that straight.el checks all packages for modifications at the beginning of a transaction, because this is faster than checking each package individually. So if you don't wrap in a transaction, then you're checking all 320 packages for modifications 320 times. O(n²) makes everybody cry.

Addendum: if you look in your profiling results, it's clear that straight--cache-package-modifications is the bottleneck, as predicted by this analysis.

Addendum 2: any questions about the transaction system? Or suggestions on how I can make the important considerations more obvious in the documentation?

[raxod502]

Also, food for thought. Would anybody be interested in me making a pull request to Spacemacs to add an option for using straight.el instead of package.el (much like there is an option to use helm vs. ivy or evil vs. the One True Keybindings)?

[vyp]

@raxod502 This thread may interest you in that case (or maybe not 🤷‍♀️): quelpa/quelpa#93 (comment)

[raxod502]

Created syl20bnr/spacemacs#9505.

[emacs18]

Thanks for all the feedback. I found out why things were taking so long.
It was because I launched emacs via

emacs -Q -l .../init-file-for-straight.el

If I do not use -Q and setup my ~/.emacs to do the equivalent thing, then the startup
time goes from 15 minutes down to 4 seconds. Now that I know this, I'll just avoid using -Q, and instead mess with my ~/.emacs to setup straight.

[raxod502]

See also this snippet from the user manual, which explains the proper way to load straight.el using emacs -Q -l .../init-file-for-straight.el.

There is one final user-facing note about the transaction system, which is important when you want to load your init-file after Emacs init has already completed, but before straight.el has been loaded (so you cannot just wrap the call in straight-transaction). To cover this edge case (which arises, for example, when you wish to profile your init-file using something like esup), you should use the following pattern:

(unwind-protect
    (let ((straight-treat-as-init t))
      "load your init-file here")
  (straight-finalize-transaction))

[braham-snyder]

@emacs18 any chance you want to share your most recent versions of my-straight-installed-p etc., and perhaps indicate where within your user-init-file / dotspacemacs-filepath you've put it all?

[raxod502]

Closing this issue since we have a Gitter for general discussion now. For requesting a better-supported way to use straight.el and package.el together with use-package, open a new issue.

[raxod502]

@rgrinberg Note that using package.el and straight.el together is now trivial with the new version of use-package, since straight.el uses the new :straight keyword while package.el continues to use the old :ensure keyword, and they can be combined.

[bradyt]

Does that mean straight-enable-use-package-integration should be deprecated?

[raxod502]

I don't see why; can you explain?

[bradyt]

I thought most of the idea with straight-enable-use-package-integration, was to override the use of :ensure to use straight.el, instead of package.el. And later, I thought a separate keyword was added to use-package so that :ensure can now always correspond with package.el. Sorry if I misunderstand.

[raxod502]

Everything you said is correct, except that you are omitting the part where in the default (i.e. non-deprecated) configuration, straight-enable-use-package-integration defines the :straight keyword that allows you to use straight.el with use-package. That part is definitely not deprecated as without it there is no way to use the packages together. You can read more in the docstrings of the relevant variables.