{"slug": "named-entity-recognition-using-scikit-llm", "title": "Named Entity Recognition Using Scikit-LLM", "summary": "Scikit-LLM, the open-source library from developer Iryna Kondr, now supports custom named entity recognition through its GPTExplainableNER estimator, letting users define entity dictionaries of key-value pairs that serve as prompts for an LLM instead of fine-tuning on large labeled datasets. The walkthrough runs entirely in Google Colab against Groq's free-tier API, redirecting the library's OpenAI-standard client via SKLLMConfig.set_gpt_url(\"https://api.groq.com/openai/v1\") and an API key stored as the Colab secret GROQ_API_KEY, then extracts EXECUTIVE, CORPORATION, PRODUCT and CITY entities from three sample tech-business sentences. The approach matters because it replaces traditional NER training pipelines with prompt-driven extraction that can be pointed at any OpenAI-compatible provider.", "body_md": "# Named Entity Recognition Using Scikit-LLM\n\nBuild standardized custom NER pipelines in a few simple steps, powered by the reasoning capabilities of LLMs.\n\n## Introduction\n\n**Structured data extraction** from text has plenty of uses in data analysis and myriad applications, including building predictive machine learning models. **Named entity recognition** (NER) is a clear example of this: it consists of transforming unstructured, messy text into structured data as a result of identifying, extracting, and categorizing relevant information.\n\nNER has been traditionally conducted using traditional natural language processing (NLP) libraries, but these models usually require huge training datasets and arduous fine-tuning to make them learn to recognize custom entities. Thankfully, **[Scikit-LLM](https://github.com/iryna-kondr/scikit-llm)** arose to bring large language model (LLM) reasoning capabilities to the table and help build standardized machine learning pipelines in a few simple steps, all on a variety of target tasks beyond just text classification, including custom NER.\n\n## Running Named Entity Recognition with Scikit-LLM\n\nThis example can be run entirely in a Google Colab notebook. It uses **[Groq](https://console.groq.com/keys)**'s LLM API to fully rely on free models. A prerequisite is to access the Groq website, register, and generate your own API key. Once generated, carefully copy your API key (you can do this only once) and paste it as a new \"Secret\" variable in Google Colab — found at the key icon on the left-hand side of Colab. Here, you'll paste your copied API key and name the secret variable `GROQ_API_KEY`.\n\nAfter that, we start our notebook by installing Scikit-LLM:\n\n```\n!pip install scikit-llm -q\n```\n\nNow it's time to dive into Python code, and the first stage is to initialize Scikit-LLM configuration. This library is designed using OpenAI API standards, but it lets us override the base URL with the `SKLLMConfig.set_gpt_url()` method, letting us point to any other compatible provider, such as Groq:\n\n``` python\nfrom google.colab import userdata\nfrom skllm.config import SKLLMConfig\n\n# Securely retrieving the Groq API key from Colab Secrets\ngroq_key = userdata.get('GROQ_API_KEY')\n\n# Redirecting scikit-LLM to Groq's high-speed, free-tier compatible endpoint\nSKLLMConfig.set_gpt_url(\"https://api.groq.com/openai/v1\")\nSKLLMConfig.set_openai_key(groq_key)\n```\n\nNext, we define the data for our example use case. Unlike conventional text classification in which we have class labels associated with text examples, a central piece of data in LLM-driven NER is the **entity dictionary**: a list of key-value pairs where keys represent categories to extract and values are the entity descriptions, which will serve as prompts or instructions to guide our LLM through the reasoning process.\n\nBesides the entities, we also define a small set of example texts related to the tech business domain and closely aligned with our entity dictionary:\n\n```\n# Defining the domain-specific entity set to extract, including a clear description that serves as instruction for the LLM\nentities = {\n    \"EXECUTIVE\": \"The full name of a company leader or individual.\",\n    \"CORPORATION\": \"The name of a technology company or business.\",\n    \"PRODUCT\": \"A specific hardware device, software, or technology feature.\",\n    \"CITY\": \"A geographical location or municipality.\"\n}\n\n# Unstructured text data to parse\nX_unstructured = [\n    \"Satya Nadella announced Copilot integrations for Microsoft at the summit in Seattle.\",\n    \"Elon Musk supervised the rollout of the Cybertruck at the Texas facility.\",\n    \"Jensen Huang showcased the new Blackwell architecture for Nvidia in San Jose.\"\n]\n```\n\nHere comes the critical step: initializing and running a Scikit-LLM estimator that will wrap a compatible pre-trained model from our chosen provider (Groq). For NER, the exact class to use is called `GPTExplainableNER`. As of the time of writing, a compatible free-tier model to use alongside this object is `openai/gpt-oss-20b`. Once the model has been initialized and wrapped into the task-specific Scikit-LLM object along with the custom entities, `fit_transform()` triggers the reasoning process of extracting structured entities from raw texts.\n\n``` python\nfrom skllm.models.gpt.tagging.ner import GPTExplainableNER\n\n# Initializing the NER estimator using a tool-compatible model\nner = GPTExplainableNER(\n    entities=entities, \n    model=\"custom_url::openai/gpt-oss-20b\" \n)\n\n# fit_transform processes the text and extracts the structured entities\nextracted_data = ner.fit_transform(X_unstructured)\n```\n\nTo show the results, we iterate over the raw texts and the extracted outputs returned by the LLM as follows:\n\n``` python\nimport ast\n\n# Displaying the extracted entities and handle the string-serialized output\nfor text, data in zip(X_unstructured, extracted_data):\n    print(f\"Original: '{text}'\")\n    \n    # If the model returned a string representation of the dict, evaluate it\n    if isinstance(data, str):\n        try:\n            data = ast.literal_eval(data)\n        except (ValueError, SyntaxError):\n            # Fallback just in case the LLM hallucinated the formatting\n            print(f\"  Raw unstructured output: {data}\\n\" + \"-\" * 50)\n            continue\n            \n    # Data is a Python dictionary, allowing iteration through .items()\n    for entity_type, extractions in data.items():\n        print(f\"  {entity_type}: {extractions}\")\n    print(\"-\" * 50)\n```\n\nResults:\n\n```\nOriginal: 'Satya Nadella announced Copilot integrations for Microsoft at the summit in Seattle.'\n  Raw unstructured output: Satya Nadella is the full name of a well‑known company leader, specifically the CEO of Microsoft, which fits the definition of an EXECUTIVE.EXECUTIVESatya Nadella announced Copilot is a specific software feature developed by Microsoft, qualifying it as a PRODUCT.PRODUCTCopilot integrations for Microsoft is the name of a technology company, matching the definition of a CORPORATION.CORPORATIONMicrosoft at the summit in Seattle is a recognized municipality in the United States, fitting the definition of a CITY.CITYSeattle.\n--------------------------------------------------\nOriginal: 'Elon Musk supervised the rollout of the Cybertruck at the Texas facility.'\n  Raw unstructured output: The name \"Elon Musk\" refers to a well‑known individual who holds a leadership position in a company (CEO of Tesla). According to the definition, an EXECUTIVE is the full name of a company leader or individual, so \"Elon Musk\" fits this category.EXECUTIVEElon Musk supervised the rollout of the The term \"Cybertruck\" is the name of a specific hardware product developed by Tesla. The definition of PRODUCT is a specific hardware device, software, or technology feature, which matches \"Cybertruck\".PRODUCTCybertruck at the Texas facility.\n--------------------------------------------------\nOriginal: 'Jensen Huang showcased the new Blackwell architecture for Nvidia in San Jose.'\n  Raw unstructured output: The name \"Jensen Huang\" refers to the CEO of Nvidia, a well‑known company leader. According to the definition, an EXECUTIVE is the full name of a company leader or individual.EXECUTIVEJensen Huang showcased the new The phrase \"Blackwell architecture\" denotes a specific GPU architecture released by Nvidia, which is a hardware technology feature. This matches the definition of a PRODUCT.PRODUCTBlackwell architecture for The word \"Nvidia\" is the name of a technology company. The definition of CORPORATION is the name of a technology company or business.CORPORATIONNvidia in The location \"San Jose\" is a city in California, fitting the definition of a CITY as a geographical location or municipality.CITYSan Jose.\n--------------------------------------------------\n```\n\nThe raw outputs are not especially clean. To view a more thoroughly parsed result, the following helper code uses regular expressions to parse and post-process the outputs into a neatly structured format:\n\n``` python\nimport re\n\ndef parse_extracted_data(raw_data):\n    parsed_entities = {}\n    # Find all entity blocks\n    entity_blocks = re.findall(r\"(.*?)\", raw_data, re.DOTALL)\n    for block in entity_blocks:\n        tag_match = re.search(r\"(.*?)\", block)\n        value_match = re.search(r\"(.*?)\", block)\n        if tag_match and value_match:\n            entity_type = tag_match.group(1)\n            entity_value = value_match.group(1)\n            if entity_type not in parsed_entities:\n                parsed_entities[entity_type] = []\n            parsed_entities[entity_type].append(entity_value)\n    return parsed_entities\n\n# Displaying the extracted entities in a cleaner format\nprint(\"\\n--- Cleaned Extracted Entities ---\")\nfor i, (text, raw_output) in enumerate(zip(X_unstructured, extracted_data)):\n    print(f\"Original Text {i+1}: '{text}'\")\n    cleaned_data = parse_extracted_data(raw_output)\n    if cleaned_data:\n        for entity_type, extractions in cleaned_data.items():\n            print(f\"  {entity_type}: {', '.join(extractions)}\")\n    else:\n        print(\"  No entities found.\")\n    print(\"-\" * 50)\n```\n\nResult:\n\n```\n--- Cleaned Extracted Entities ---\nOriginal Text 1: 'Satya Nadella announced Copilot integrations for Microsoft at the summit in Seattle.'\n  EXECUTIVE: Satya Nadella\n  PRODUCT: Copilot\n  CORPORATION: Microsoft\n  CITY: Seattle\n--------------------------------------------------\nOriginal Text 2: 'Elon Musk supervised the rollout of the Cybertruck at the Texas facility.'\n  EXECUTIVE: Elon Musk\n  PRODUCT: Cybertruck\n--------------------------------------------------\nOriginal Text 3: 'Jensen Huang showcased the new Blackwell architecture for Nvidia in San Jose.'\n  EXECUTIVE: Jensen Huang\n  PRODUCT: Blackwell architecture\n  CORPORATION: Nvidia\n  CITY: San Jose\n```\n\nOur NER pipeline, despite its simplicity in setup, did a pretty good job.\n\n## Wrapping Up\n\nThis article illustrated how to use Scikit-LLM to leverage pre-trained, open-source LLMs for named entity recognition (NER), a natural language processing task whereby raw, messy texts are analyzed to extract structured data based on domain-specific entities. With minimal configuration effort and a few simple code steps, we can yield more than reasonable results without the need for training a model from scratch.\n\n \n\n \n\n[**\\[Iván Palomares Carrascosa\\](https://www.linkedin.com/in/ivanpc/)**](https://www.linkedin.com/in/ivanpc/) is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.", "url": "https://wpnews.pro/news/named-entity-recognition-using-scikit-llm", "canonical_source": "https://www.kdnuggets.com/named-entity-recognition-using-scikit-llm", "published_at": "2026-10-21 14:00:00+00:00", "updated_at": "2026-10-21 14:41:38.347956+00:00", "lang": "en", "topics": ["natural-language-processing", "large-language-models", "ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["Scikit-LLM", "Iryna Kondr", "Groq", "Google Colab", "GPTExplainableNER", "OpenAI", "Microsoft", "Nvidia"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/named-entity-recognition-using-scikit-llm", "markdown": "https://wpnews.pro/news/named-entity-recognition-using-scikit-llm.md", "text": "https://wpnews.pro/news/named-entity-recognition-using-scikit-llm.txt", "jsonld": "https://wpnews.pro/news/named-entity-recognition-using-scikit-llm.jsonld"}}