AI on demand: opendatalab/MinerU2.5-2509-1.2B: Difference between revisions

From MediaWiki
Jump to navigation Jump to search
Tag: Manual revert
No edit summary
Line 76: Line 76:
}
}
</syntaxhighlight>
</syntaxhighlight>
== Calling the model - PDF text extraction ==
Extracting text from a PDF via the API requires a few additional steps.
This is a pure text extraction from a PDF. The model does not analyse the content. 
<syntaxhighlight lang="bash">
# Set your personal key:
STONEY_KEY=sk-...
# Set the file path to desired pdf:
FILE_PATH='path/to/file/document.pdf'
# Set the desired model:
MODEL='opendatalab/MinerU2.5-2509-1.2B'
#Set where to save the extracted text.
OUTPUT_DIR='/home/user/Desktop/results'
#Call the model and extract text
(
 
  PREFIX="$OUTPUT_DIR/mineru-$(date +%Y%m%d-%H%M%S)-$$"
  # Convert PDF pages into images.
  pdftoppm -r 150 -png "$FILE_PATH" "$PREFIX"
  for PAGE in "$PREFIX"-*.png; do
    # Encode the page and submit it to MinerU.
    base64 -w 0 < "$PAGE" |
    jq -Rs --arg model "$MODEL" '{
      model: $model,
      messages: [{
        role: "user",
        content: [{
          type: "image_url",
          image_url: {url: ("data:image/png;base64," + .)}
        }]
      }],
      max_tokens: 4096,
      temperature: 0.2,
      top_p: 0.9
    }' |
    curl --silent --fail-with-body --show-error \
      --connect-timeout 5 --max-time 180 \
      https://llm.stoney-cloud.com/v1/chat/completions \
      --header "Authorization: Bearer $STONEY_KEY" \
      --header 'Content-Type: application/json' \
      --data-binary @- \
      --output "${PAGE%.png}.json"
  done
  # Combine page results and sum token usage.
  jq -s '{
    model: .[0].model,
    pages: map({
      id,
      text: .choices[0].message.content,
      finish_reason: .choices[0].finish_reason,
      usage
    }),
    usage: {
      prompt_tokens: (map(.usage.prompt_tokens) | add),
      completion_tokens: (map(.usage.completion_tokens) | add),
      total_tokens: (map(.usage.total_tokens) | add)
    },
    extraction_complete: all(.[]; .choices[0].finish_reason == "stop")
  }' "$PREFIX"-*.json > "$OUTPUT_DIR/result.json"
  jq -r '.pages[].text' "$OUTPUT_DIR/result.json" \
    > "$OUTPUT_DIR/document.txt"
  jq . "$OUTPUT_DIR/result.json"
)


[[Category: AI on Demand]]
[[Category: AI on Demand]]
[[Category: General purpose models]]
[[Category: General purpose models]]

Revision as of 10:05, 12 November 2026

Calling the model

# Set your personal key:
STONEY_KEY=sk-...

# Set the desired document to run OCR against:
FILE_PATH=document.png

# Set the desired model:
MODEL=opendatalab/MinerU2.5-2509-1.2B

# Encode document contents with Base64 and detect MIME type:
FILE_CONTENT_BASE64=$(base64 -w 0 < "$FILE_PATH")
FILE_MIME_TYPE=$(file --brief --mime-type "$FILE_PATH")

curl https://llm.stoney-cloud.com/v1/chat/completions \
        --silent --fail --show-error \
        --header "Authorization: Bearer $STONEY_KEY" \
        --header 'Content-Type: application/json' \
        --data '{
                "model": "'"$MODEL"'",
                "messages": [{
                        "role": "user",
                        "content": [{
                                "type": "image_url",
                                "image_url": {
                                        "url": "data:'"$FILE_MIME_TYPE"';base64,'"$FILE_CONTENT_BASE64"'"
                                }
                        }]
                }],
                "max_tokens": 4096,
                "temperature": 0.2,
                "top_p": 0.9
        }' \
        | jq

Example output:

{
  "id": "chatcmpl-834171fda5956922",
  "object": "chat.completion",
  "created": 1788347006,
  "model": "opendatalab/MinerU2.5-2509-1.2B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "This is a lot of 12 point text to test theOCR code and see if it works on all types of file format. The quick brown dog jumped over the lazy fox. The quick brown dog jumped over the lazy fox. The quick brown dog jumped over the lazy fox. The quick brown dog jumped over the lazy fox.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": [],
        "reasoning": null,
        "reasoning_content": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null,
      "token_ids": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 412,
    "total_tokens": 480,
    "completion_tokens": 68,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "kv_transfer_params": null
}

Calling the model - PDF text extraction

Extracting text from a PDF via the API requires a few additional steps.

This is a pure text extraction from a PDF. The model does not analyse the content.

<syntaxhighlight lang="bash">

  1. Set your personal key:

STONEY_KEY=sk-...

  1. Set the file path to desired pdf:

FILE_PATH='path/to/file/document.pdf'

  1. Set the desired model:

MODEL='opendatalab/MinerU2.5-2509-1.2B'

  1. Set where to save the extracted text.

OUTPUT_DIR='/home/user/Desktop/results'

  1. Call the model and extract text

(

 PREFIX="$OUTPUT_DIR/mineru-$(date +%Y%m%d-%H%M%S)-$$"
 # Convert PDF pages into images.
 pdftoppm -r 150 -png "$FILE_PATH" "$PREFIX"
 for PAGE in "$PREFIX"-*.png; do
   # Encode the page and submit it to MinerU.
   base64 -w 0 < "$PAGE" |
   jq -Rs --arg model "$MODEL" '{
     model: $model,
     messages: [{
       role: "user",
       content: [{
         type: "image_url",
         image_url: {url: ("data:image/png;base64," + .)}
       }]
     }],
     max_tokens: 4096,
     temperature: 0.2,
     top_p: 0.9
   }' |
   curl --silent --fail-with-body --show-error \
     --connect-timeout 5 --max-time 180 \
     https://llm.stoney-cloud.com/v1/chat/completions \
     --header "Authorization: Bearer $STONEY_KEY" \
     --header 'Content-Type: application/json' \
     --data-binary @- \
     --output "${PAGE%.png}.json"
 done
 # Combine page results and sum token usage.
 jq -s '{
   model: .[0].model,
   pages: map({
     id,
     text: .choices[0].message.content,
     finish_reason: .choices[0].finish_reason,
     usage
   }),
   usage: {
     prompt_tokens: (map(.usage.prompt_tokens) | add),
     completion_tokens: (map(.usage.completion_tokens) | add),
     total_tokens: (map(.usage.total_tokens) | add)
   },
   extraction_complete: all(.[]; .choices[0].finish_reason == "stop")
 }' "$PREFIX"-*.json > "$OUTPUT_DIR/result.json"
 jq -r '.pages[].text' "$OUTPUT_DIR/result.json" \
   > "$OUTPUT_DIR/document.txt"
 jq . "$OUTPUT_DIR/result.json"

)