Generative Model Preparation
===============================================================================

.. meta::
   :description: Learn how to use Hugging Face Hub and Optimum Intel APIs to
                 prepare generative models for inference.



Since generative AI models tend to be big and resource-heavy, it is advisable to
optimize them for efficient inference. This article will show how to prepare
LLM models for inference with OpenVINO by:

* `Downloading Models from Hugging Face <#download-generative-models-from-hugging-face-hub>`__
* `Downloading Models from Model Scope <#download-generative-models-from-model-scope>`__
* `Converting and Optimizing Generative Models <#convert-and-optimize-generative-models>`__



Download Generative Models From Hugging Face Hub
###############################################################################

Pre-converted and pre-optimized models are available in the `OpenVINO Toolkit <https://huggingface.co/OpenVINO>`__
organization, under the `model section <https://huggingface.co/OpenVINO#models>`__, or under
different model collections:

* `LLM: <https://huggingface.co/collections/OpenVINO/llm-6687aaa2abca3bbcec71a9bd>`__
* `Speech-to-Text <https://huggingface.co/collections/OpenVINO/speech-to-text-672321d5c070537a178a8aeb>`__
* `Speculative Decoding Draft Models <https://huggingface.co/collections/OpenVINO/speculative-decoding-draft-models-673f5d944d58b29ba6e94161>`__

You can also use the **huggingface_hub** package to download models:

.. code-block:: console

   pip install huggingface_hub
   huggingface-cli download "OpenVINO/phi-2-fp16-ov" --local-dir model_path


The models can be used in OpenVINO immediately after download. No dependencies
are required except **huggingface_hub**.


Download Generative Models From Model Scope
###############################################################################

To download models from `Model Scope <https://www.modelscope.cn/home>`__,
use the **modelscope** package:

.. code-block:: console

   pip install modelscope
   modelscope download --model "Qwen/Qwen2-7b" --local_dir model_path

Models downloaded via Model Scope are available in Pytorch format only and they must
be :doc:`converted to OpenVINO IR <../../openvino-workflow/model-preparation/convert-model-to-ir>`
before inference.

Convert and Optimize Generative Models
###############################################################################

OpenVINO works best with models in the OpenVINO IR format, both in full precision and quantized.
If your selected model has not been pre-optimized, you can easily do it yourself, using a single
**optimum-cli** command. For that, make sure optimum-intel is installed on your system:

.. code-block:: console

   pip install optimum-intel[openvino]


While optimizing models, you can decide to keep the original precision or select one that is lower.

.. tab-set::

   .. tab-item:: Keeping full model precision
      :sync: full-precision

      .. code-block:: console

         optimum-cli export openvino --model <model_id> --weight-format fp16 <exported_model_name>

      Examples:

      .. tab-set::

         .. tab-item:: LLM (text generation)
            :sync: llm-text-gen

            .. code-block:: console

               optimum-cli export openvino --model meta-llama/Llama-2-7b-chat-hf --weight-format fp16 ov_llama_2

         .. tab-item:: Diffusion models (text2image)
            :sync: diff-text-img

            .. code-block:: console

               optimum-cli export openvino --model stabilityai/stable-diffusion-xl-base-1.0 --weight-format fp16 ov_SDXL

         .. tab-item:: VLM (Image processing):
            :sync: vlm-img-proc

            .. code-block:: console

               optimum-cli export openvino --model openbmb/MiniCPM-V-2_6 --trust-remote-code –weight-format fp16 ov_MiniCPM-V-2_6

         .. tab-item:: Whisper models (speech2text):
            :sync: whisp-speech-txt

            .. code-block:: console

               optimum-cli export openvino --trust-remote-code --model openai/whisper-base ov_whisper

         .. tab-item:: SpeechT5 TTS models (text2speech):
            :sync: whisp-speech-txt

            .. code-block:: console

               optimum-cli export openvino --model microsoft/speecht5_tts --model-kwargs "{\"vocoder\": \"microsoft/speecht5_hifigan\"}" ov_speecht5_tts

   .. tab-item:: Exporting to selected precision
      :sync: low-precision

      .. code-block:: console

         optimum-cli export openvino --model <model_id> --weight-format int4 <exported_model_name>

      Examples:

      .. tab-set::

         .. tab-item:: LLM (text generation)
            :sync: llm-text-gen

            .. code-block:: console

               optimum-cli export openvino --model meta-llama/Llama-2-7b-chat-hf --weight-format int4 ov_llama_2

         .. tab-item:: Diffusion models (text2image)
            :sync: diff-text-img

            .. code-block:: console

               optimum-cli export openvino --model stabilityai/stable-diffusion-xl-base-1.0 --weight-format int4 ov_SDXL

         .. tab-item:: VLM (Image processing)
            :sync: vlm-img-proc

            .. code-block:: console

               optimum-cli export openvino -m model_path --task text-generation-with-past --weight-format int4 ov_MiniCPM-V-2_6


.. note::

   Any other ``model_id``, for example ``openbmb/MiniCPM-V-2_6``, or the path
   to a local model file can be used.

   Also, you can specify different data type like ``int8``.


Additional Resources
###############################################################################

* `Full set of optimum-cli parameters <https://huggingface.co/docs/optimum/en/intel/openvino/export>`__
* :doc:`Model conversion in OpenVINO <../../openvino-workflow/model-preparation/convert-model-to-ir>`
* :doc:`Model optimization in OpenVINO <../../openvino-workflow/model-optimization>`
