Google's LangExtract on a Custom OpenAI-Compatible API: Model IDs That Route to Ollama, Strict Schemas and Silent Chunks

LangExtract is Google's Python library for pulling structured entities out of long text, with every extraction mapped back to its character position in the source. You describe the task, give one or two examples, and it chunks the document, sends each chunk to a model, parses the JSON and aligns the results. Gemini is the default, but it has an OpenAI provider built on the official OpenAI SDK, so it can run against vLLM, LiteLLM or a hosted gateway.

Getting it to use that provider with a non-OpenAI model name is the first surprise. I tested langextract 1.7.1 (openai 3.35.1, Python 3.13) against a small local mock server that logged every request and could return 400, 429, 503, fenced JSON or truncated output on demand. Everything below is what went over the wire. The examples point at APIClaw, an OpenAI-compatible gateway I build, so weigh my opinions accordingly. Its base URL is https://apiclaw.biz/v1; replace YOUR_MODEL_ID with an exact ID from your provider's model list.

The obvious call is:

import langextract as lx

result = lx.extract(
    text_or_documents=text,
    prompt_description=prompt,
    examples=examples,
    model_id="qwen3-coder-plus",
    api_key="your-key",
    model_url="https://apiclaw.biz/v1",
)

LangExtract routes on regex patterns over model_id. Only names starting with gpt-3.5, gpt-4, gpt-5 or o1 to o9 go to the OpenAI provider. Names starting with qwen, deepseek, llama, mistral, gemma, phi and gpt-oss, among others, go to the Ollama provider. Here is what happened in my tests:

qwen3-coder-plus and gpt-oss-120b went to the Ollama client, which POSTed to /v1/api/generate on my base URL and failed with Can't find Ollama qwen3-coder-plus. Try: ollama run qwen3-coder-plus. plain-model, failed with No provider registered for model_id='plain-model'. gpt-4o-mini with model_url worked and hit /v1/chat/completions. So the code above only works if your gateway's model happens to be named like an OpenAI model.

import os
import langextract as lx

config = lx.factory.ModelConfig(
    model_id="YOUR_MODEL_ID",
    provider="OpenAILanguageModel",   # "openai" also worked
    provider_kwargs={
        "api_key": os.environ["APICLAW_API_KEY"],
        "base_url": "https://apiclaw.biz/v1",
        "temperature": 0.0,
        "max_output_tokens": 2000,
        "max_workers": 4,
    },
)

result = lx.extract(
    text_or_documents=text,
    prompt_description="Extract every person mentioned, with their role. Use exact text.",
    examples=examples,
    config=config,
    show_progress=False,
)
for e in result.extractions:
    print(e.extraction_class, e.extraction_text, e.attributes, e.char_interval)

That sent POST /v1/chat/completions for every model name I tried. Two details:

api_key explicitly.OPENAI_API_KEY from the environment when the model ID contains gpt. With an explicit provider and a name like API key not provided. even though base_url can come from the environment.OPENAI_BASE_URL. Keep /v1 on the end either way. You'll also see a warning: With 'config', schema constraints are still applied via examples. That matters for the next section.

Every chunk became one request with:

You are a helpful assistant that responds in JSON format. Q:/ A: pairs with the expected JSON, and the chunk text,n: 1, response_format of type json_schema, named langextract_extractions, with strict: true, built from your examples. By default there was no temperature and no max_tokens, so the server's defaults applied. Setting temperature and max_output_tokens in provider_kwargs sent temperature: 0.0 and max_tokens: 2000. For extraction I'd always set both: a low temperature for consistency and an output cap that fits a long list of entities.

Strict json_schema output is not universal. Some models and some OpenAI-compatible servers support only json_object, or nothing. My mock returned a 400 for json_schema, and the whole extract() call raised InferenceRuntimeError: OpenAI API error: Error code: 400.

result = lx.extract(..., config=config, use_schema_constraints=False)

With use_schema_constraints=False the request switched to response_format: {"type": "json_object"} and succeeded. The examples in the prompt still show the model the expected shape.

This is the setting I'd change first. By default LangExtract suppresses parse errors per chunk. Each of these runs printed one log line and returned zero extractions with no exception:

Here is the result: followed by a fenced json block). Under json_object mode LangExtract expects raw JSON, so the chunk was omitted: Chunk omitted from results (parse error): Failed to parse JSON content. finish_reason: "length". The truncated JSON failed to parse and the chunk was dropped the same way. On a 40-chunk document, a few dropped chunks look like the model just didn't find anything there. Two fixes:

result = lx.extract(
    ...,
    config=config,
    use_schema_constraints=False,
    fence_output=True,                                  # accept fenced json replies
    resolver_params={"suppress_parse_errors": False},   # fail loudly instead
)

fence_output=True parsed the fenced reply correctly. suppress_parse_errors: False turned the silent drop into a ResolverParsingError, which you can catch and retry. A reply that started with a <think>...</think> block parsed fine in my test, so reasoning models weren't a problem there.

I ran a 5,320-character document with the defaults:

Each request repeats your prompt description and examples, so with small chunks the fixed part can be a large share of every request. max_workers defaults to 10, which means a long document can fire 10 requests at once. On a plan with a per-minute or concurrency limit, set it lower. extraction_passes runs the whole document again per pass, one pass after another, so three passes triple both requests and time.

A larger max_char_buffer cuts request count, but long chunks give the model more to track, so test recall on your own documents before raising it a lot.

A server that always returned 503 got three requests (the SDK's default two retries) and then extract() raised InferenceRuntimeError. A 429 with Retry-After: 2 was honoured: the SDK waited and the third request succeeded after about 4.6 seconds. LangExtract adds no retry loop of its own, so for big batch jobs wrap the call in your own retry, and remember that a failure in one chunk fails the whole document.

ModelConfig(provider="OpenAILanguageModel") unless your model ID starts with gpt- or o1 to o9. provider_kwargs; the env key is only read for temperature and max_output_tokens yourself.use_schema_constraints=False if your server rejects suppress_parse_errors: False (and fence_output=True for chatty models) so dropped chunks don't go unnoticed.max_workers to fit your rate limit, and count extraction_passes as a multiplier. Disclosure: I build APIClaw, the OpenAI-compatible gateway in these examples (flat monthly plans; crypto on the site, other payment methods through support). Everything above applies to any compatible endpoint.