xemantic-kotlin-script-python

Stateful Python, embedded with Jep, in Kotlin scripts

Why?

xemantic-kotlin-script runs Kotlin snippets in stateful sessions, with diagnostics an LLM can act on. Some tasks are better served by the Python ecosystem (numpy, pandas, scikit-learn, …), so this project embeds a CPython interpreter into a Kotlin script session, keeping the Python state between the snippets as well.

Requirements

Jep is a native CPython extension: the Java library comes from Maven Central, but its native part comes with the jep Python package, which has to be installed in the Python environment in use:

pip install jep==4.3.2

The version of the jep Python package must match the version of the org.ninia:jep Java library. Building it requires a JDK, found through JAVA_HOME.

The Python executable is, in this order:

1. PythonEnvironment.pythonExecutable, if set before the first use of Python,
2. the one of the virtual environment of the VIRTUAL_ENV environment variable,
3. python3 on the PATH.

The Python code imports the modules from the same paths as this executable, e.g. the site-packages of a virtual environment, but not the ones of its base Python.

Usage

In build.gradle.kts add:

dependencies {
    implementation("com.xemantic.kotlin:xemantic-kotlin-script-python:0.1.0")
}

Then:

Python().use { python ->
    ScriptSession {
        "python" provides python
    }.use { session ->
        session.execute("""python.exec("import numpy as np; a = np.arange(10)")""")
        session.execute("""python["factor"] = 3""")
        val result = session.execute("""python.eval<List<Long>>("(a * factor).tolist()")""")
        println((result as ScriptResult.Value).value) // [0, 3, 6, 9, 12, 15, 18, 21, 24, 27]
    }
}

The examples assume numpy installed in the Python environment, otherwise the code can declare it as a dependency, see below.

• A Python is stateful: the global variables, functions, classes and imports of the executed code stay visible to the code executed after it, also across the snippets of a ScriptSession.
• exec(code) executes Python code, eval<T>(expression) evaluates an expression converted to T, and python["name"] reads or writes a global variable. The values are converted by Jep, e.g. a Python int becomes a Long, a str a String, and a list a List.
• python(code) executes Python code like a cell of a Jupyter notebook, returning the value of its last statement, if it is an expression, or null otherwise, so that the code written by an LLM can either provide a result, or be executed only for its effects:

  python("import numpy as np; a = np.arange(10)") // null
  python("a.sum()")                                 // 45L

  The value is converted by Jep to its default Java type, and a value which Jep cannot convert is returned as its Python repr.
• For an AI agent, the Python can be started lazily, only when a script uses it, as the first one initializes the Python environment:

  class Agency : AutoCloseable {
      private val lazyPython = lazy { Python() }
      val python: Python by lazyPython
      override fun close() {
          if (lazyPython.isInitialized()) python.close()
      }
  }

  With "agency" provides agency, a script can then call agency.python("..."), or set agency.python["name"] = value.
• Each Python has its own global variables, while the imported modules are shared by all of them, so that native extension modules, like numpy, can be used.
• A Jep interpreter can be used only by the thread which created it, so each Python runs its code on a thread of its own, and can be called from any thread.
• A ScriptSession does not close its dependencies, close the Python when no longer needed.
• The code passed to exec, or python(code), can declare its dependencies, like a @file:DependsOn of a Kotlin snippet, with inline script metadata, as with uv run:

  # /// script
  # dependencies = ["rdkit"]
  # ///
  from rdkit import Chem

  The missing dependencies are installed before executing the code, with uv if it is on the PATH, or with pip of the Python executable otherwise, into PythonEnvironment.dependencyDirectory, by default the directory of the xemantic.kotlin.script.python.dependencyDirectory system property, if set, or otherwise ~/.cache/xemantic-kotlin-script-python/<python>-<hash of the environment>, which becomes importable once any code declares its dependencies. The installed dependencies are shared by all the Python interpreters of the JVM, and they cannot change the versions of the distributions of the Python environment itself, nor the ones already imported from the dependency directory, as a module cannot be replaced once imported. A dependency which cannot be installed fails the call with PythonDependencyException, with the output of the installer.
• A Python exception fails the call with jep.JepException, reported by the ScriptSession as an ExecutionPhase.EVALUATION error, like:

  jep.JepException: <class 'ValueError'>: boom
    at <string>.<module>(<string>:1)
  

Development

The tests run in the .venv Python virtual environment of the project, created by Gradle with python3, with only the jep package of the version of the library installed, like a clean environment of an agent. The tests declare the Python packages they use, like numpy, as dependencies, installed from PyPI into build/python-dependencies, which Gradle cleans before every test run, so the tests need network access.

See DEVELOPMENT.md for the documentation conventions, and for maintenance notes on updating the gradle wrapper and the project dependencies.