GGUF (Breaking Change to Model Files)

[abetlen]

GGUF support for llama.cpp Closes #628

Currently works to update your old ggml v3 llama models run

python3 vendor/llama.cpp/convert-llama-ggmlv3-to-gguf.py --input <path-to-ggml> --output <path-to-gguf>

TODO

• Fix tests
• Move convert script into package to make it easier for people to migrate Add docs link to conversion script in llama.cpp
• Fix detokenization bug (getting extra leading space)

[sndani]

Hello, excited about trying this out with the CodeLlama gguf model.

Followed the MacOS (Sonoma beta) instructions. How do I get the 'llama' shared library?

llama-cpp-python % python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1
Traceback (most recent call last):
....
File "... /llama-cpp-python/llama_cpp/llama_cpp.py", line 80, in
_lib = _load_shared_library(_lib_base_name)
File " ... /llama-cpp-python/llama_cpp/llama_cpp.py", line 71, in _load_shared_library
raise FileNotFoundError(
FileNotFoundError: Shared library with base name 'llama' not found

Thanks!

[abetlen]

@sndani try reinstalling with the --verbose flag, it's likely a build error and will be reported by cmake. If the issue isn't resolvable please open an issue and I'll take a look. Cheers

[sndani]

@abetlen thanks for the great work and thanks for responding.
Yes, did run
% CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dir --verbose

Turns out, cmake isn't getting to target libllama.so under vendor/llama.cpp (but 'make clean' try deleting it). This is a dev fix but will open an issue (or the next person who encounters this can) if this isn't only my environment for some reason.

% make clean
% cd vendor/llama.cpp
% LLAMA_METAL=on make libllama.so
% cd ../..
% pip install 'llama-cpp-python[server]'
% export LLAMA_CPP_LIB=./vendor/llama.cpp/libllama.so
% python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1

[reddiamond1234]

My model is now a lot slower... is there any solution to fix this?

[abdullrahmanesmael07]

Hi, i always get this error.

C:\Users\abdul\OneDrive\Desktop\Coding playground\python\LLM\llama-cpp-python>python -m pip install -e .
Obtaining file:///C:/Users/abdul/OneDrive/Desktop/Coding%20playground/python/LLM/llama-cpp-python
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions>=4.5.0 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from llama_cpp_python==0.3.1) (4.12.2)
Requirement already satisfied: numpy>=1.20.0 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from llama_cpp_python==0.3.1) (1.26.4)
Requirement already satisfied: diskcache>=5.6.1 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from llama_cpp_python==0.3.1) (5.6.3)
Requirement already satisfied: jinja2>=2.11.3 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from llama_cpp_python==0.3.1) (3.1.3)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from jinja2>=2.11.3->llama_cpp_python==0.3.1) (2.1.5)
Building wheels for collected packages: llama_cpp_python
  Building editable for llama_cpp_python (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for llama_cpp_python (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      *** scikit-build-core 0.10.7 using CMake 3.29.0 (editable)
      *** Configuring CMake...
      2024-10-17 10:21:25,535 - scikit_build_core - WARNING - Can't find a Python library, got libdir=None, ldlibrary=None, multiarch=None, masd=None
      loading initial cache file C:\Users\abdul\AppData\Local\Temp\tmp9my5wqdi\build\CMakeInit.txt
      -- Building for: NMake Makefiles
      CMake Error at CMakeLists.txt:3 (project):
        Running

         'nmake' '-?'

        failed with:

         no such file or directory


      CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
      CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
      -- Configuring incomplete, errors occurred!

      *** CMake configuration failed
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for llama_cpp_python
Failed to build llama_cpp_python
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (llama_cpp_python)

[cccampos123]

Hi abdullrahmanesmael07, if you are using windows, install visual studio community with c++ options and reboot. It will works.