Run DeepSeek locally at home on

How to run Deepseek model locally using Jetson orin nx board

Because of memory constraints, Q4_K_S run on orin nx
https://huggingface.co/TheBloke/deeps...

$ jetson_release -v
Software part of jetson-stats 4.3.1 - (c) 2024, Raffaello Bonghi
Model: NVIDIA Jetson Orin NX Engineering Reference Developer Kit - Jetpack 6.2 [L4T 36.4.3]
NV Power Mode[2]: 15W
Serial Number: [XXX Show with: jetson_release -s XXX]
Hardware:
699-level Part Number: 699-13767-0001-300 R.1
P-Number: p3767-0001
Module: NVIDIA Jetson Orin NX (8GB ram)
SoC: tegra234
CUDA Arch BIN: 8.7
Platform:
Machine: aarch64
System: Linux
Distribution: Ubuntu 22.04 Jammy Jellyfish
Release: 5.15.148-tegra
Python: 3.10.12
jtop:
Version: 4.3.1
Service: Active
Libraries:
CUDA: 12.6.68
cuDNN: 9.3.0.75
TensorRT: 10.3.0.30
VPI: 3.2.4
Vulkan: 1.3.204
OpenCV: 4.8.0 - with CUDA: NO

sudo apt install python3-pip git cmake
pip install huggingface-hub

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build
cmake .. DLLAMA_CUDA=1
make

cd ..
mkdir models
cd models
huggingface-cli download TheBloke/deepseek-llm-7B-chat-GGUF deepseek-llm-7b-chat.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False

cd ../build
./bin/llama-cli -m ../models/deepseek-llm-7b-chat.Q4_K_M.gguf --interactive --gpu-layers 999 -c 2048