Introduction
Immediate tuning is a way that makes use of frozen pre-trained language fashions to downstream duties that decrease per-task storage and reminiscence utilization through the coaching part and that is helpful for Massive Language Fashions (LLMs) similar to GPT2, T5, GPT-J, GPT-NEO, GPT-NEOX, GPT-20B, GPT3, and so on the place the mannequin is so massive that fine-tuning turns into tough or very costly.
A pre-trained language mannequin parameters are frozen and solely embedding parameters for a specific process are up to date through the coaching part.
The outline of the duty is included within the precise enter sentence not directly. The duty description is named immediate because it prompts the mannequin to carry out a selected process.
Advantages of Immediate Tuning
The dimensions of huge pre-trained language fashions is consistently rising. Whereas fine-tuning these fashions would require extra reminiscence, fine-tuned fashions additionally get bigger, so it could be prohibitively costly to retailer and infer a fine-tuned copy of the mannequin for every downstream process. Whereas prompt-tuning would require much less reminiscence and the scale of immediate tuned mannequin can be much less in comparison with the fine-tuned mannequin.
Disadvantage of Immediate Tuning
Immediate tuning takes extra coaching time than fine-tuningThe designing of the prompts in order that the mannequin can perceive
Right here on this tutorial, we are going to discover ways to infer the prompt-tuned mannequin for sentiment evaluation. For that you will want a prompt-tuned mannequin, to carry out immediate tuning you’ll be able to verify our colab file Immediate Tuning Massive Language Mannequin. When you get the immediate tuned mannequin, you’ll be able to comply with the under steps to carry out the inference.
Useful resource Necessities
Colab Professional:25 GB RAM2 x vCPUT4 GPU
Earlier than you infer prompt-tuned mannequin, it’s essential to be sure that all of the required packages are put in in your system. Carry out the next steps to put in all of the required packages:
Set up Apex
git clone
cd apex
git checkout nm_v1.11.0
pip set up -v –disable-pip-version-check –no-cache-dir –global-option=”–cpp_ext” –global-option=”–cuda_ext” –global-option=”–fast_layer_norm” –global-option=”–distributed_adam” –global-option=”–deprecated_fused_adam” ./
Set up Nemo toolkit
apt-get replace && apt-get set up -y libsndfile1 ffmpeg
pip set up Cython
pip set up nemo_toolkit[‘all’]
Obtain the gpt mannequin’s .nemo file.
After getting put in the packages, now we are going to obtain the mannequin. On this tutorial, we’re utilizing nemo_gpt1.3B_fp16.nemo massive mannequin for immediate tuning, you’ll be able to obtain the nemo_gpt1.3B_fp16.nemo as proven under:
wget
Clone the Nemo repo from GitHub
git clone
cd NeMo/examples/nlp/language_modeling
Inference
To carry out the inference, run the megatron_gpt_prompt_learning_eval.py file as proven under.
python megatron_gpt_prompt_learning_eval.py
+virtual_prompt_model_file=PATH_TO_NEMO_PROMPT_LEARNING_MODEL_FILE
gpt_model_file=PATH_TO_FROZEN_GPT_MODEL_FILE
inference.grasping=True
inference.add_BOS=False
coach.gadgets=1
coach.num_nodes=1
tensor_model_parallel_size=1
pipeline_model_parallel_size=1
pred_file_path=PATH_WHERE_PRED_TEXT_FILE_WILL_BE_SAVED
data_paths=[path/to/prompt_1.json, path/to/prompt_2.json]
+virtual_prompt_model_file = Path to your immediate tuned .nemo filegpt_model_file = Path to the gpt mannequin’s .nemo filedata_paths = Listing of json or jsonl recordsdata that comprises the testing enter
Observe: The json file ought to have keys that match the fields specified within the immediate template, which is used through the coaching part.
For instance, the immediate template through the coaching part appears to be like like under:
“<|VIRTUAL_PROMPT_0|> {sentence} sentiment: {label}”
{“taskname”: “sentiment”, “sentence”: “some textual content”}
Enter
The pattern json file appears to be like just like the one belowPrompt_1.json
{“taskname”: “sentiment”, “sentence”: “The film was not good.”},
Output
On operating the inference, it would create a textual content file that comprises the outcomes. Beneath is the pattern textual content file.outcomes.txt
The film was not good. sentiment: destructive
