How can I use dynamic schemas?

[anhnmt]

I have a proto file and gprc client address localhost:8080

How can I connect to gprc client using the above proto file without having to generate .pb.go or .connect.go files?

This mechanism of action may look like this: https://docs.konghq.com/hub/kong-inc/grpc-gateway/how-to/

Also I quite enjoy using this library, thank you for creating this wonderful library 😁

[emcfarlane]

Hey @anhnmt thanks for trying out the library! For dynamic protobuf files you can create the service with vanguard.NewServiceWithSchema. See here for details on how to dynamically load protobufs.

To clarify is it a gRPC server on localhost:8080? To create a reverse proxy you could setup the proxy server as:

var schema protoreflect.ServiceDescriptor // Loaded dynamically
remote, _ := url.Parse("http://localhost:8080")
proxy := httputil.NewSingleHostReverseProxy(remote)
transcoder := vanguard.NewTranscoder(
	[]*vanguard.Service{
		 vanguard.NewServiceWithSchema(schema, proxy),
	},
)

[jhump]

@anhnmt, hi! Glad you are finding this repo valuable!

For a little more info on obtaining descriptors, to use with vanguard.NewServiceWithSchema, you'll do something like this:

1. Read the configured schema file. A schema file will usually be a binary-encoded google.protobuf.FileDescriptorSet message. You'll unmarshal the file contents into a new message of that type (*descriptorpb.FileDescriptorSet in the Go runtime). From there, you can create a *protoregistry.Files using protodesc.NewFiles and then create a type resolver using dynamicpb.NewTypes.
   • Note you could also retrieve the schema using gRPC Reflection or a Buf Schema Registry.
2. Find the relevant services using files.FindDescriptorByName. You can then type-assert the result to a protoreflect.ServiceDescriptor and provide to vanguard.NewServiceWithSchema.
   • You'll also want to provide the type resolver as an option, via vanguard.WithTypeResolver(types). This allows for the same schema file to be consulted when marshaling and un-marshaling extensions and google.protobuf.Any messages.

[anhnmt]

@emcfarlane I use httputil.NewSingleHostReverseProxy but it seems it only reverses HTTP proxies, gRPC does not.
Do you have any method to reverse gRPC proxy?

[jhump]

@anhnmt, gRPC uses HTTP under the hood. If you are having an issue, it is likely due to use of HTTP 1.1, whereas gRPC requires HTTP/2. If you are not using TLS (where the HTTP/2 protocol can be negotiated during the TLS handshake), then you need to be using "HTTP/2 over clear text", also called "H2C". There is more detail in the Connect docs, since this is also necessary to use the gRPC protocol with the connect-go module: https://connectrpc.com/docs/go/deployment/#h2c

In particular, you'd need to wrap your reverse proxy handler using h2c.NewHandler as in the code snippet in that link, if your proxy server accepts gRPC requests without TLS. And you need to set the Transport field of the httputil.ReverseProxy to a transport implementation supports H2C. The link above shows an example of creating an http2.Transport that works over plaintext.

[anhnmt]

@jhump, I used h2c but it seems I made a mistake in some step so I still can't connect to gRPC
Here is my source code: https://github.com/anhnmt/gprc-dynamic-proto/blob/main/cmd/dynamic/main.go

[emcfarlane]

Hey @anhnmt, I think the reverse proxy should be created with a http2 transport here:

proxy := httputil.NewSingleHostReverseProxy(target)
proxy.Transport = &http2.Transport{AllowHTTP: true}
h2cHandler := h2c.NewHandler(proxy, &http2.Server{})

Let me know if this works for you. You may need to configure the transport for your server configuration.

[anhnmt]

@emcfarlane

proxy.Transport = &http2.Transport{AllowHTTP: true}

I also tried your method but encountered some other problems, for example:

• REST: 502 Bad Gateway
• gRPC: 14 UNAVAILABLE

[emcfarlane]

        proxy := httputil.NewSingleHostReverseProxy(target)
+       proxy.Transport = &http2.Transport{
+               AllowHTTP: true,
+               DialTLS: func(network, addr string, _ *tls.Config) (net.Conn, error) {
+                       // If you're also using this client for non-h2c traffic, you may want
+                       // to delegate to tls.Dial if the network isn't TCP or the addr isn't
+                       // in an allowlist.
+                       return net.Dial(network, addr)
+               },
+       }
        h2cHandler := h2c.NewHandler(proxy, &http2.Server{})

Tested with curl localhost:8000/v1/users/1 and looks like it works!

[anhnmt]

@emcfarlane
Thank you very much 😁
I also found a similar way here:

vanguard-go/internal/examples/pets/cmd/pets-fe/main.go

Lines 66 to 71 in d7faf70

	proxy.Transport = h2cIfGRPCTransport{h2c: &http2.Transport{
	AllowHTTP: true,
	DialTLSContext: func(ctx context.Context, network, addr string, _ *tls.Config) (net.Conn, error) {
	return (&net.Dialer{}).DialContext(ctx, network, addr)
	},
	}}

Also, can you give me some feedback on optimizing this part?

import (
	"bytes"
	"fmt"
	"io"
	"net/http"
	"net/http/httputil"
	"net/url"
	"os"
	"path/filepath"
	"time"

	"connectrpc.com/vanguard"
	"github.com/jhump/protoreflect/desc/protoparse"
	"github.com/rs/zerolog"
	"github.com/rs/zerolog/log"
	"golang.org/x/net/http2"
	"golang.org/x/net/http2/h2c"
	"google.golang.org/protobuf/reflect/protodesc"
	"google.golang.org/protobuf/types/descriptorpb"
	"google.golang.org/protobuf/types/dynamicpb"
)

	googleapis := []string{
		"google/api/annotations.proto",
		"google/api/http.proto",
		"google/protobuf/descriptor.proto",
	}

	files := append(googleapis, "user/v1/user.proto")

	p := protoparse.Parser{
		ImportPaths: []string{
			"proto",
			"googleapis",
		},
		Accessor: func(filename string) (io.ReadCloser, error) {
			return ReadFileContent(filename)
		},
	}

	fds, err := p.ParseFiles(files...)
	if err != nil {
		log.Err(err).Msg("could not parse given files")
		return
	}

	fileDescriptors := make([]*descriptorpb.FileDescriptorProto, 0)
	for _, fd := range fds {
		fileDescriptors = append(fileDescriptors, fd.AsFileDescriptorProto())
	}

	newFiles, err := protodesc.NewFiles(&descriptorpb.FileDescriptorSet{
		File: fileDescriptors,
	})
	if err != nil {
		log.Err(err).Msg("could not parse given files")
		return
	}

	types := dynamicpb.NewTypes(newFiles)

	name, err := newFiles.FindDescriptorByName("user.v1.UserService")
	if err != nil {
		return
	}
	serviceDesc := name.ParentFile().Services().ByName("UserService")

[emcfarlane]

The use of protoparse is really interesting! If you want to parse the descriptors up front you could use buf's images. See the docs here. Then build the image up front and include the filedescriptor sets dynamically:buf build -o users.binpb. @jhump will have better advice!

A bug I noticed in the proto. The option in users.proto:

    option (google.api.http) = {get: "/v1/users/{page=**}"};

Should be {page=*}(or just {page}) as ** will never be able to be unmarshalled into the page int32 field. There does look like an issue with stacking multiple Transcoder instances and error handling.

1. Directly curling the server: curl localhost:8080/v1/users/1/123

{"code":3,"message":"invalid parameter \"page\" invalid character '/' after top-level value","details":[]}

2. Proxying the server: curl localhost:8000/v1/users/1/123

{"code":2, "message":"response uses incorrect codec: expecting \"json\" but instead got \"?\"", "details":[]}

[jhump]

I saw the use of protocompile in the commented out code. Was there a reason you're using the older protoparse instead? The result of protocompile is a slice of protoreflect.FileDescriptor instances, but the slice is a named type (linker.Files) that also has a method AsResolver(), so you can use that to look up descriptors by name. That way no conversion or registry construction are necessary.

In the existing code, which does convert descriptors and construct a registry (using protodesc.NewFiles), one strange thing here is that when converting from *desc.FileDescriptor -> protoreflect.FileDescriptor, the code re-builds the protoreflect.FilDescriptor from the underlying proto. But that isn't necessary if you are using v1.15+ of github.com/jhump/protoreflect/desc/protoparse. You can instead just use fileDesc.UnwrapFile(). So creating a resolver with all of them could instead look like so:

resolver := &protoregistry.Files{}
for _, fileDesc := range parsedFiles {
    if err := resolver.RegisterFile(fileDesc.UnwrapFile()); err != nil {
        return err
    }
}

(Having said that, using protocompile would be even better.)

As @emcfarlane said, you could use an image or file descriptor set file, which may be simpler to distribute to production containers than all of the source needed to compile the schema. And if you push the code to the BSR, then you could instead download the schema via a reflection endpoint. In fact, there's a Go library that includes the ability to watch a schema in the BSR, downloading a new version when one becomes available (and, under the hood, it uses that reflection endpoint).

[anhnmt]

@jhump, I used protocompile and got the following error, while github.com/jhump/protoreflect did not get this case

My source code: https://github.com/anhnmt/gprc-dynamic-proto/blob/fc77aa12da23fbaaa171e97b4814c8185ba5ab05/cmd/protocompile/main.go

panic: invalid type: got *dynamicpb.Message, want *annotations.HttpRule

goroutine 1 [running]:
google.golang.org/protobuf/internal/impl.(*messageConverter).GoValueOf(0xc00050bb40, {{}, 0xdef640?, 0xc0004a9bc0?, 0xedee28?})
        C:/Users/ANHNMT/go/pkg/mod/google.golang.org/[email protected]/internal/impl/convert.go:457 +0x408
google.golang.org/protobuf/internal/impl.(*ExtensionInfo).InterfaceOf(0x12a1ee0?, {{}, 0xdef640?, 0xc0004a9bc0?, 0x12a1f00?})
        C:/Users/ANHNMT/go/pkg/mod/google.golang.org/[email protected]/internal/impl/extension.go:102 +0x5a
google.golang.org/protobuf/proto.GetExtension({0xed7140?, 0xc00009a9c0?}, {0xedee28, 0x12a1ee0})
        C:/Users/ANHNMT/go/pkg/mod/google.golang.org/[email protected]/proto/extension.go:45 +0x9b
connectrpc.com/vanguard.getHTTPRuleExtension({0xee0e88?, 0xc0002022a0?})
        C:/Users/ANHNMT/go/pkg/mod/connectrpc.com/[email protected]/protocol_http.go:342 +0x66
connectrpc.com/vanguard.(*Transcoder).registerMethod(0xc000202480, {0xed6360?, 0xc00052a8d0}, {0xee0e88, 0xc0002022a0}, 0xc0004a9c40)
        C:/Users/ANHNMT/go/pkg/mod/connectrpc.com/[email protected]/transcoder.go:204 +0x2e5
connectrpc.com/vanguard.(*Transcoder).registerService(0xc000202480, 0xc0003b5ee8, {{0xedc538, 0xc0000403f0}, 0xc000539f20, 0xc000539ec0, 0xc000539ef0, {0xe09b31, 0x5}, 0xffffffff, ...})
        C:/Users/ANHNMT/go/pkg/mod/connectrpc.com/[email protected]/transcoder.go:127 +0x6f8
connectrpc.com/vanguard.NewTranscoder({0xc0003b5e58, 0x1, 0xb?}, {0x0, 0x0, 0x6?})
        C:/Users/ANHNMT/go/pkg/mod/connectrpc.com/[email protected]/vanguard.go:131 +0x628
main.main()
        C:/Golang/gprc-dynamic-proto/cmd/dynamic/main.go:149 +0x6bc

Process finished with the exit code 2

[jhump]

@anhnmt, ah, right. Sorry, I forgot about the HTTP annotations. It is indeed expected that they are dynamic extensions from the compiler (since the compiler's version of the message definition could differ from the version linked into the calling program).

Seems like a good addition to protocompile would be a helper to address that, or maybe even a compiler option so that you don't have to do anything as a post-process. The helper or option would basically do the same thing that protoparse does, which is to re-parse custom options using protoregistry.GlobalTypes as the resolver.