Support protoc Kotlin code generation

[PhilKes]

Closes #39127

This PR adds support to generate Kotlin DSLs/Extensions for protoc via adding the kotlin_out parameter to the protoc command:

• By default, the Kotlin code generation is enabled if the dependency io.quarkus:quarkus-kotlin is present
• It can be explicetely en-/disabled via the quarkus.generate-code.grpc.kotlin.generate property

To test it I reused the kotlin-grpc-project integration-test to use the generated DSL for HelloMsg.
Now currently when running the KotlinGRPCProjectBuildTest, compileKotlin fails with:

e: file://.../quarkus/integration-tests/gradle/target/classes/kotlin-grpc-project/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:17 Annotation argument must be a compile-time constant.
e: file://.../quarkus/integration-tests/gradle/target/classes/kotlin-grpc-project/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:37 Unresolved reference 'kotlin'.
e: file://.../quarkus/integration-tests/gradle/target/classes/kotlin-grpc-project/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:16:24 Unresolved reference 'kotlin'.

HelloMsgKt.kt

// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: hello.proto

// Generated files should ignore deprecation warnings
@file:Suppress("DEPRECATION")
package io.quarkus.example;

@kotlin.jvm.JvmName("-initializehelloMsg")
public inline fun helloMsg(block: io.quarkus.example.HelloMsgKt.Dsl.() -> kotlin.Unit): io.quarkus.example.HelloMsg =
io.quarkus.example.HelloMsgKt.Dsl._create(io.quarkus.example.HelloMsg.newBuilder()).apply { block() }._build()
/**
* Protobuf type `io.quarkus.example.HelloMsg`
*/
public object HelloMsgKt {
@kotlin.OptIn(com.google.protobuf.kotlin.OnlyForUseByGeneratedProtoCode::class)
@com.google.protobuf.kotlin.ProtoDslMarker
public class Dsl private constructor(
  private val _builder: io.quarkus.example.HelloMsg.Builder
) {
  public companion object {
    @kotlin.jvm.JvmSynthetic
    @kotlin.PublishedApi
    internal fun _create(builder: io.quarkus.example.HelloMsg.Builder): Dsl = Dsl(builder)
  }

  @kotlin.jvm.JvmSynthetic
  @kotlin.PublishedApi
  internal fun _build(): io.quarkus.example.HelloMsg = _builder.build()

  /**
   * `string message = 1;`
   */
  public var message: kotlin.String
    @JvmName("getMessage")
    get() = _builder.getMessage()
    @JvmName("setMessage")
    set(value) {
      _builder.setMessage(value)
    }
  /**
   * `string message = 1;`
   */
  public fun clearMessage() {
    _builder.clearMessage()
  }

  /**
   * `.google.protobuf.Timestamp date_time = 2;`
   */
  public var dateTime: com.google.protobuf.Timestamp
    @JvmName("getDateTime")
    get() = _builder.getDateTime()
    @JvmName("setDateTime")
    set(value) {
      _builder.setDateTime(value)
    }
  /**
   * `.google.protobuf.Timestamp date_time = 2;`
   */
  public fun clearDateTime() {
    _builder.clearDateTime()
  }
  /**
   * `.google.protobuf.Timestamp date_time = 2;`
   * @return Whether the dateTime field is set.
   */
  public fun hasDateTime(): kotlin.Boolean {
    return _builder.hasDateTime()
  }

  /**
   * `.io.quarkus.example.HelloMsg.Status status = 3;`
   */
  public var status: io.quarkus.example.HelloMsg.Status
    @JvmName("getStatus")
    get() = _builder.getStatus()
    @JvmName("setStatus")
    set(value) {
      _builder.setStatus(value)
    }
  public var statusValue: kotlin.Int
    @JvmName("getStatusValue")
    get() = _builder.getStatusValue()
    @JvmName("setStatusValue")
    set(value) {
      _builder.setStatusValue(value)
    }
  /**
   * `.io.quarkus.example.HelloMsg.Status status = 3;`
   */
  public fun clearStatus() {
    _builder.clearStatus()
  }
}
}
@kotlin.jvm.JvmSynthetic
public inline fun io.quarkus.example.HelloMsg.copy(block: `io.quarkus.example`.HelloMsgKt.Dsl.() -> kotlin.Unit): io.quarkus.example.HelloMsg =
`io.quarkus.example`.HelloMsgKt.Dsl._create(this.toBuilder()).apply { block() }._build()

public val io.quarkus.example.HelloMsgOrBuilder.dateTimeOrNull: com.google.protobuf.Timestamp?
get() = if (hasDateTime()) getDateTime() else null

I am guessing this is because the generated kotlin code should not be in build/classes/java but rather build/classes/kotlin?
But if we provide a different output path for the kotlin_out than for java_out we would need a way to add this extra path also to the compile sources, which is done in CodeGenerator.init(), right?

[github-actions]

🙈 The PR is closed and the preview is expired.

[gsmet]

This looks promising, thanks for the PR.

I suggested some minor doc changes and more importantly it seems some Gradle tests are failing and I think that's the important bit:

2024-11-17T17:13:44.2059374Z Options for KOTLIN DAEMON: IncrementalCompilationOptions(super=CompilationOptions(compilerMode=INCREMENTAL_COMPILER, targetPlatform=JVM, reportCategories=[0, 3], reportSeverity=2, requestedCompilationResults=[0], kotlinScriptExtensions=[]), areFileChangesKnown=false, modifiedFiles=null, deletedFiles=null, classpathChanges=NotAvailableForNonIncrementalRun, workingDir=/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/cacheable, multiModuleICSettings=MultiModuleICSettings(buildHistoryFile=/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/local-state/build-history.bin, useModuleDetection=false), usePreciseJavaTracking=true, icFeatures=IncrementalCompilationFeatures(withAbiSnapshot=false, preciseCompilationResultsBackup=true, keepIncrementalCompilationCachesInMemory=true, enableUnsafeIncrementalCompilationForMultiplatform=false), outputFiles=[/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/kotlin/main, /home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/cacheable, /home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/local-state])
2024-11-17T17:13:44.2061194Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:17 Annotation argument must be a compile-time constant.
2024-11-17T17:13:44.2063191Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:37 Unresolved reference 'kotlin'.
2024-11-17T17:13:44.2065030Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:16:24 Unresolved reference 'kotlin'.
2024-11-17T17:13:44.2065363Z Finished executing kotlin compiler using DAEMON strategy

[gsmet]

@cdsap I'm not sure if it's related to this change but I have seen the following in the logs and it doesn't look like something we would want:

2024-11-17T17:13:44.2065823Z 2 problems were found storing the configuration cache.
2024-11-17T17:13:44.2068318Z - Task :quarkusDev of type io.quarkus.gradle.tasks.QuarkusDev: cannot serialize object of type 'org.gradle.api.internal.artifacts.configurations.DefaultUnlockedConfiguration', a subtype of 'org.gradle.api.artifacts.Configuration', as these are not supported with the configuration cache.
2024-11-17T17:13:44.2069077Z See https://docs.gradle.org/8.9/userguide/configuration_cache.html#config_cache:requirements:disallowed_types
2024-11-17T17:13:44.2071428Z - Task :quarkusDev of type io.quarkus.gradle.tasks.QuarkusDev: cannot serialize object of type 'org.gradle.api.internal.tasks.DefaultSourceSet', a subtype of 'org.gradle.api.tasks.SourceSet', as these are not supported with the configuration cache.
2024-11-17T17:13:44.2072168Z See https://docs.gradle.org/8.9/userguide/configuration_cache.html#config_cache:requirements:disallowed_types

[cdsap]

@gsmet I tested with a version without the configuration cache changes (3.15.1), I'm observing the same log:
https://ge.solutions-team.gradle.com/s/g33okmapudxeg/console-log?page=1#L73

Note that I'm executing the build like: ./gradlew clean quarkusDev --configuration-cache --no-build-cache

Will confirm with the team

[PhilKes]

This looks promising, thanks for the PR.

I suggested some minor doc changes and more importantly it seems some Gradle tests are failing and I think that's the important bit:

2024-11-17T17:13:44.2059374Z Options for KOTLIN DAEMON: IncrementalCompilationOptions(super=CompilationOptions(compilerMode=INCREMENTAL_COMPILER, targetPlatform=JVM, reportCategories=[0, 3], reportSeverity=2, requestedCompilationResults=[0], kotlinScriptExtensions=[]), areFileChangesKnown=false, modifiedFiles=null, deletedFiles=null, classpathChanges=NotAvailableForNonIncrementalRun, workingDir=/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/cacheable, multiModuleICSettings=MultiModuleICSettings(buildHistoryFile=/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/local-state/build-history.bin, useModuleDetection=false), usePreciseJavaTracking=true, icFeatures=IncrementalCompilationFeatures(withAbiSnapshot=false, preciseCompilationResultsBackup=true, keepIncrementalCompilationCachesInMemory=true, enableUnsafeIncrementalCompilationForMultiplatform=false), outputFiles=[/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/kotlin/main, /home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/cacheable, /home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/local-state])
2024-11-17T17:13:44.2061194Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:17 Annotation argument must be a compile-time constant.
2024-11-17T17:13:44.2063191Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:37 Unresolved reference 'kotlin'.
2024-11-17T17:13:44.2065030Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:16:24 Unresolved reference 'kotlin'.
2024-11-17T17:13:44.2065363Z Finished executing kotlin compiler using DAEMON strategy

Thanks for the feedback. I am aware of the failing tests, as I described in the PR description I am not sure what the problem is since I dont have much experienced with mixed java/kotlin generated sources. Do the generated kotlin sources have to be in another folder than the generated java code? (see my original PR description)

[PhilKes]

Any feedback on this, @geoand, @alesj, @cescoffier? 🙂

[cescoffier]

The PR looks good, but the Gradle issue is problematic.

[cescoffier]

@PhilKes did you look at the gradle issue?

[PhilKes]

@PhilKes did you look at the gradle issue?

@cescoffier I looked at it, the generated HelloMsgKt needs the additional dependency protobuf-kotlin for some annotations:

// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: hello.proto
//...
public object HelloMsgKt {
  @kotlin.OptIn(com.google.protobuf.kotlin.OnlyForUseByGeneratedProtoCode::class)
  @com.google.protobuf.kotlin.ProtoDslMarker
  public class Dsl private constructor(
 //...

Now I am not sure how to proceed, should I add a check for the protobuf-kotlin dependency being present (like containsQuarkusKotlin()) or should protobuf-kotlin be added to quarkus-kotlin dependencies?

[cescoffier]

Should protobuf-kotlin be added to quarkus-kotlin dependencies

Does it need to be available at runtime or only at build time? If it's only build time, we can "just" document that when using Gradle this dependency is necessary.

[PhilKes]

Should protobuf-kotlin be added to quarkus-kotlin dependencies

Does it need to be available at runtime or only at build time? If it's only build time, we can "just" document that when using Gradle this dependency is necessary.

Indeed it's only needed at build-time so I added it as a compileOnly dependency and added a note in the documentation

[cescoffier]

Just a minor comment on the doc

[cescoffier]

Thanks!

[cescoffier]

Let's see what the CI says.

[PhilKes]

@cescoffier looks like there is something wrong with the build pipeline?

./mvnw: line 327: /c/hostedtoolcache/windows/Java_Temurin-Hotspot_jdk/17.0.14-7/x64/bin/java: Argument list too long
https://github.com/quarkusio/quarkus/actions/runs/13529572158/job/37808886665?pr=44552#step:17:36

[cescoffier]

Hum, no, on Windows we need to use the file approach to workaround that issue. So, arguments are written in a file and passed as @file if I remember correctly

[quarkus-bot]

Status for workflow Quarkus Documentation CI

This is the status report for running Quarkus Documentation CI on commit 9199d38.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

Warning

There are other workflow runs running, you probably need to wait for their status before merging.

[quarkus-bot]

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit 9199d38.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

You can consult the Develocity build scans.

[PhilKes]

Hum, no, on Windows we need to use the file approach to workaround that issue. So, arguments are written in a file and passed as @file if I remember correctly

Well the build succeeded now after an update via ebase

[cescoffier]

Thanks!

[edeandrea]

Hello. I think this is the culprit of what we're seeing in #23612 (comment)

In the superheroes sample application, the grpc-locations application build is now failing since March 2.

Setting quarkus.generate-code.grpc.kotlin.generate=false does allow me to get around this problem, but there is definitely a problem here.

I've created #46675 to track this.