fix failure of llava/pixtral

[sywangyi]

What does this PR do?

• tokenizers: @ArthurZucker and @itazap

[sywangyi]

from transformers import  LlavaForConditionalGeneration, AutoProcessor

model_id = "mistral-community/pixtral-12b"
processor = AutoProcessor.from_pretrained(model_id)
text = "Describe the images"
inputs = processor.tokenizer(text)
print(f"Input text: '{text}'")
print(f"Token IDs: {inputs['input_ids']}")
decoded_text = processor.tokenizer.decode(inputs["input_ids"])
print(f"Decoded text: '{decoded_text}'")

output "DescribeĠtheĠimages", expected output "Describe the images"(in transformers 4.57.3)

[sywangyi]

also, fix the failure in llava test

[sywangyi]

Mistral3ProcessorTest case fail because the tokenizer fix.

from transformers import AutoProcessor

model_id = "hf-internal-testing/Mistral-Small-3.1-24B-Instruct-2503-only-processor"
processor = AutoProcessor.from_pretrained(model_id)
text = "Describe the images"
inputs = processor.tokenizer(text)
print(f"Input text: '{text}'")
print(f"Token IDs: {inputs['input_ids']}")
decoded_text = processor.tokenizer.decode(inputs["input_ids"])
print(f"Decoded text: '{decoded_text}'")

before the fix
Input text: 'Describe the images'
Token IDs: [1, 5847, 13089, 1278, 8061]
Decoded text: '<s>DescribeĠtheĠimages'

after the fix
Input text: 'Describe the images'
Token IDs: [1, 5847, 1972, 22326, 1268, 8926]
Decoded text: '<s>Describetheimages'

however in 4.57.3, the output is 
Decoded text: '<s>Describe the images' which is expected, I did some investigate, seem the pre_tokenizer is incorrect for "hf-internal-testing/Mistral-Small-3.1-24B-Instruct-2503-only-processor" in 5.0.0, should be Sequence(pretokenizers=[Split(pattern=Regex("(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\n]+|\s+(?!\S)|\s+"), behavior=Isolated, invert=False), ByteLevel(add_prefix_space=False, trim_offsets=True, use_regex=False)]) instead of Metaspace(replacement="▁", prepend_scheme=always, split=False)

[sywangyi]

seems the pre_tokenizer should be loaded from tokenizer.json, but in v5.0.0, it does not do it.

[molbap]

Normally #42894 should fix tokenization issues on main and on the next release candidate. Might need some time with the holiday season now, apologies.There are a few different changes on your PR, can you put what fails in the PR description and ensure your PR fixes minimally? Thanks!

[sywangyi]

from transformers import LlavaForConditionalGeneration, AutoProcessor

model_id = "mistral-community/pixtral-12b"
processor = AutoProcessor.from_pretrained(model_id)
text = "Describe the images"
inputs = processor.tokenizer(text)
print(f"Input text: '{text}'")
print(f"Token IDs: {inputs['input_ids']}")
decoded_text = processor.tokenizer.decode(inputs["input_ids"])
print(f"Decoded text: '{decoded_text}'")

Hi, I tried #42894, but it does not fix the "hf-internal-testing/Mistral-Small-3.1-24B-Instruct-2503-only-processor" and "mistral-community/pixtral-12b" issue, The issue is that nearly all cases of pytest tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest fail, because of tokenizer issue and case issue, and I fix them.

[sywangyi]

@molbap I update the PR to fix all the issue I mentioned in the PR including llava test, tokenizer issue of mistral-community/pixtral-12b and hf-internal-testing/Mistral-Small-3.1-24B-Instruct-2503-only-processor

[yao-matrix]

@molbap , could you pls help take a second look? Thx very much.

[molbap]

Hello, thanks for the investigation. However it seems that on the mistral example #42894 indeed fixes it.

from transformers import  LlavaForConditionalGeneration, AutoProcessor

model_id = "mistral-community/pixtral-12b"
processor = AutoProcessor.from_pretrained(model_id)
text = "Describe the images"
inputs = processor.tokenizer(text)
print(f"Input text: '{text}'")
print(f"Token IDs: {inputs['input_ids']}")
decoded_text = processor.tokenizer.decode(inputs["input_ids"])
print(f"Decoded text: '{decoded_text}'")

does return

Input text: 'Describe the images'
Token IDs: [5847, 13089, 1278, 8061]
Decoded text: 'Describe the images'

If the other tests are not fixed feel free to update the PR!

[itazap]

Hey! We just merged #42894 which fixes this issue! If you have any cases that aren't resolved by this please feel free to share here :)

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: llava

[sywangyi]

Hey! We just merged #42894 which fixes this issue! If you have any cases that aren't resolved by this please feel free to share here :)

here, after updating to the latest main. the tokenizer issue I mentioned before has been resolved, but new issue pop up.

tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest::test_tokenizer_integration - AssertionError: Lists differ: ['<|im_start|>', 'sy', 'st', 'em', '\n', 'An', 'sw', 'er', ' [245 chars]'\n'] != ['<|im_start|>', 'system',...

this case failure.
see https://github.com/huggingface/transformers/blob/main/tests/models/llava/test_modeling_llava.py#L564 seems the behavior change, is it a regression?

[sywangyi]

for case fix, like dtype and device mismatch issue, fix it in case @ydshieh, please help review.

[ArthurZucker]

We indeed defaulted to auto dtype now!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

run-slow: llava

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ["models/llava"]
quantizations: []

[github-actions]

CI Results

Workflow Run ⚙️

✅ No failing test specific to this PR 🎉 !