{"slug": "faster-and-local-jev-like-model-for-mac", "title": "Faster and local Jev like model for Mac", "summary": "Laya released laya-coreml, an open-weight Core ML model that runs typed decision inference locally on Apple Silicon's Neural Engine, sustaining 49.1–50.0 decisions/s across three uncapped 600-step Snake episodes with zero deaths and two safety interventions. On an M3 Max with ANE FP16, a single multilingual decision measured 4.98 ms P50 / 5.31 ms P95 and 2.78× better whole-system energy per decision than compiled MLX FP16, while a validated W8 palette variant reached 4.88 ms and 3.19× energy improvement; the requested 10× improvement was not achieved. The ANE bundle caps requests at a 96-token total, with the general-purpose 1024-token model available as aac6fef/laya-multilingual-coreml, and following upstream v0.3.5 calibration temperatures are clamped to [0.5, 5.0].", "body_md": "**Open-weight typed decisions on Apple Silicon. Core ML, Neural Engine, zero generated tokens.**\n\n[PyPI](https://pypi.org/project/laya-coreml/) · [Hugging Face weights](https://huggingface.co/aac6fef/laya-multilingual-coreml-ane) · [中文](https://github.com/mizorewww/laya-coreml/blob/main/README.zh-CN.md)\n\nA real Laya model plays Snake locally, with visible probabilities, score, length,\nlatency and safety interventions. The GIF replays a recorded Core ML run at **1× speed**.\nThe game uses explicit planner features and a visible cycle safety layer.\n\nThe complete active Snake loop sustained **49.1–50.0 decisions/s** across three\nuncapped 600-step episodes, with zero deaths and two safety interventions.\n[Game-loop timings and paced-rate limits](https://github.com/mizorewww/laya-coreml/blob/main/docs/SNAKE_BENCHMARKS.md)\ninclude rendering serialization; terminal painting is excluded.\n\n**One short multilingual decision: 4.98 ms P50 / 5.31 ms P95 on M3 Max with ANE FP16.**\nThe same experiment measured **2.78× better whole-system energy per decision** than\ncompiled MLX FP16. A separately validated W8 palette variant reached 4.88 ms and\n3.19× energy improvement. These are single-question results, not full Snake frame\ntimes; the requested 10× improvement was not achieved.\n\nApple Silicon · macOS 15+ · Python 3.11–3.13.\n\n```\npip install 'laya-coreml[demo]'\nhf download aac6fef/laya-multilingual-coreml-ane --local-dir models/snake\nlaya-coreml-snake --model ./models/snake\n```\n\nDownload once, then play offline. No PyTorch, Transformers or MLX is needed for inference. The terminal needs 104 columns × 35 rows. Space pauses; ↑/↓ changes speed; R resets; Q quits. First-time Core ML initialization can take tens of seconds.\n\n[Controls, recording and video export](https://github.com/mizorewww/laya-coreml/blob/main/docs/SNAKE_DEMO.md)\n· [Measured stable decision rates](https://github.com/mizorewww/laya-coreml/blob/main/docs/SNAKE_BENCHMARKS.md)\n· [Shareable video and recording provenance](https://github.com/mizorewww/laya-coreml/blob/main/docs/LAUNCH.md)\n\n```\npip install laya-coreml\npython\nimport laya_coreml as laya\n\nagent = laya.load(\"aac6fef/laya-multilingual-coreml-ane\")\nresult = agent.predict(\n    \"The customer requests a refund of a duplicate payment.\",\n    {\n        \"refund\": {\n            \"type\": \"noul\",\n            \"instructions\": \"Does the customer request a refund?\",\n        }\n    },\n)\nprint(result[\"answers\"][\"refund\"])\n```\n\nLaya returns probabilities for **choice**, ordinal **score**, and boolean **noul**\nquestions. There is no autoregressive decoding or generated JSON to parse. Hub\nmodels download before initialization; subsequent predictions stay local. Pass\n`local_files_only=True` to require an existing cache, or load a local directory.\n\nFollowing upstream v0.3.5, fitted calibration temperatures are clamped to\n`[0.5, 5.0]` before use: the shipped `choice:11+` bucket is 0.1006, which would\nsharpen logits ~10x and report a coin flip as near-certainty. The checkpoint's\nraw values remain available as `agent.temperature_raw` and\n`agent.temperature_by_options_raw`, and a `RuntimeWarning` names every clamped\nbucket at load.\n\nThe ANE bundle has a **96-token total limit**, including question, options and\nstate. Longer requests raise a capacity error. Use\n`aac6fef/laya-multilingual-coreml` for the general-purpose 1024-token model.\n[Full API, model selection and offline usage](https://github.com/mizorewww/laya-coreml/blob/main/docs/USAGE.md).\n\n40-core GPU, 128 GiB, macOS 27.2. One 91-token question padded to 96, including\nprompt preparation, tokenization, arrays, synchronous inference, calibration and\nformatting. Loading and warmup are excluded. MLX enables compile, prefix caching\nand shape buckets. Six alternating 20-second blocks per implementation produced\n**65,598 stable calls**.\n\n| Metric | Compiled MLX FP16 | Core ML ANE FP16 | Core ML ANE W8 | \n|---|---|---|---|\n| P50 / P95 | 6.94 / 7.39 ms | **4.98 / 5.31 ms** | **4.88 / 5.23 ms** | \n| Mean system power estimate | 61.39 W | 30.75 W | 27.39 W | \n| System energy / decision | 0.4288 J | **0.1540 J** | **0.1344 J** | \n| Speed gain | 1× | **1.39×** | **1.42×** | \n| System energy gain | 1× | **2.78×** | **3.19×** | \n\nEnergy uses direct SMC PSTR sensor readings, with raw samples and explicit anomaly\nrejection. This is an estimate with sensor and background-load uncertainty.\n**Speed gain × average power ratio = energy gain**; multiplying energy by speed\nagain would double-count time. The W8 variant compresses weights while retaining\nFP16 compute. It is approximate, and its package-size reduction is not a speed ratio.\n\n[Speed, energy and hardware evidence](https://github.com/mizorewww/laya-coreml/blob/main/docs/ANE_BENCHMARKS.md)\n· [Raw measurements](https://github.com/mizorewww/laya-coreml/tree/main/benchmarks/results).\n\n| Hugging Face bundle | Default engine | Capacity | Purpose | \n|---|---|---|---|\n| [Laya 421M](https://huggingface.co/aac6fef/laya-coreml) | CPU + GPU | 512 tokens | Original English model | \n| [Multilingual 322M](https://huggingface.co/aac6fef/laya-multilingual-coreml) | CPU + GPU | 1024 tokens | General multilingual decisions | \n| [Typed Decisions 421M](https://huggingface.co/aac6fef/laya-typed-decisions-coreml) | CPU + GPU | 1024 tokens | Original specialized checkpoint | \n| [Snake GPU](https://huggingface.co/aac6fef/laya-multilingual-coreml-snake) | CPU + GPU | B3 / L64 | Batches the three compact game questions | \n| [Multilingual ANE](https://huggingface.co/aac6fef/laya-multilingual-coreml-ane) | CPU + ANE | B1 / L96 | Short decisions, FP16 | \n| [Multilingual ANE W8](https://huggingface.co/aac6fef/laya-multilingual-coreml-ane-w8) | CPU + ANE | B1 / L96 | Optional approximate palette compression | \n\nEvery bundle includes tokenizer/configuration, model card, provenance, checksums and packaging-time validation. ANE bundles also include the exact original host embedding/action tensors they need. No original training checkout is required.\n\nThe three general-purpose FP16 checkpoints match upstream selected answers on\n**189/189 validation questions**. Each passes 100 repeated calls. ANE FP16 L96\npasses **59/59 fitting questions**, with maximum calibrated-probability drift\n0.002925; W8 passes the same subset with drift 0.014393 under an unchanged 0.02\ngate. Six- and four-bit experiments failed that gate and are not published weights.\nThese are conversion-fidelity fixtures, not proof of general task accuracy.\n\nA separately exported FP16 ANE L1024 graph passes the complete **63/63** fixture,\nbut an actual 1024-token request takes about **91.7 ms** in its serial screen.\nThe short ANE result does not establish a long-context advantage. A 600-step\npaired Snake check matches **600/600 actions**, with zero deaths and zero shield\ninterventions; the current ANE adapter's three sequential calls do not establish\na consistent full-game speedup over compiled MLX.\n\nThe ordinary SDPA Core ML export and the ANE graph are different implementations. The ordinary export defaults to CPU+GPU after unrestricted RangeDim GPU shapes failed local fidelity checks. Changing its device setting alone does not reproduce the ANE result. The ANE rewrite uses BC1L activations, 1×1 projections and per-head attention; its plan and a separate Instruments trace support Neural Engine work. CPU still handles input/output boundaries.\n\nTo export yourself, install `laya-coreml[convert]` and run\n`laya-coreml convert laya-multilingual models/custom`. ANE research conversion,\ncompression and benchmark scripts live in the Git checkout. The inference wheel\ncontains the portable runtime and optional terminal demo.\n\nApache-2.0. Independent port of [Laya](https://github.com/NandhaKishorM/laya), by\nConvai Innovations and contributors, building on the [MLX sibling project](https://github.com/mizorewww/laya-mlx).\nNot an official Convai Innovations or Apple release. See\n[NOTICE](https://github.com/mizorewww/laya-coreml/blob/main/NOTICE).", "url": "https://wpnews.pro/news/faster-and-local-jev-like-model-for-mac", "canonical_source": "https://github.com/mizorewww/laya-coreml", "published_at": "2026-09-22 18:20:21+00:00", "updated_at": "2026-09-22 18:54:18.193944+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "machine-learning", "ai-products"], "entities": ["Laya", "laya-coreml", "Apple Silicon", "Core ML", "Neural Engine", "M3 Max", "MLX", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/faster-and-local-jev-like-model-for-mac", "markdown": "https://wpnews.pro/news/faster-and-local-jev-like-model-for-mac.md", "text": "https://wpnews.pro/news/faster-and-local-jev-like-model-for-mac.txt", "jsonld": "https://wpnews.pro/news/faster-and-local-jev-like-model-for-mac.jsonld"}}