Protobuf is commonly used to refer to both the serialization format and the IDL. You can use protos to specify JSON/REST endpoints. In past companies I've also used it to specify DB schemas.
> while Smithy is just an idl used to describe services
What's the use of such descriptions aside from diagrams? I realize there's e.g. terraform that use similar kind of language to describe what to create/destroy.
The main use case is to auto generate clients for services and even create stubs for service implementations.
AWS sdks need to be implemented in dozens of languages so something like Smithy helps in code gen for multiple languages, avoiding the massively manual task of creating client apis.
Amazon has done this internally across several different iterations and it's fantastic. Being able to write a service definition and object model and then generating clients and service stubs makes it easy to model what client/server interaction will look like apart from implementation. When I was there, there were at least 3 protocol representations that you could expose for the same model, make it simple to support a variety of use cases.
gRPC to my knowledge uses protocol buffers to define services. But it's more of a framework for doing RPC.
Smithy is a much more abstract notation for defining services independent of any implementation details, and it is resource based rather than message based.
Protobuf and gRPC are great, and AWS will continue to make sure developers can be successful using them with AWS. I’ll try to explain how we ended up at Smithy instead of using other existing tools.
We started working on Smithy around 2018 because we wanted to improve the scale of our API program and the AWS SDK team to deal with the growing number of services (over 250 now!) and languages we want to support in official AWS SDKs (like the newly released Rust SDK). We had a ton of existing services that we needed to be compatible with, but we also wanted to add new features to improve new services going forward too.
We needed a very flexible meta-model that allows us to continue to evolve the model to account for things like integrating with other systems and to model service-specific customizations that each AWS SDK team can implement independently. Smithy's meta-model is based on traits, a self-describing way to add more information to models. Lots of validation can be built in to custom traits, which helps to ensure that service teams are using traits properly and adhere to their specifications. Smithy's resource modeling helps us here too because it allows AWS service teams, as they adopt Smithy, to essentially automatically support CloudFormation resource schemas. Resources also help us to point service teams in the right direction to make their services work well over HTTP (which methods to use, URIs, safety, idempotency, etc).
We needed an integrated model validation, linting, and diff tool to keep services consistent and detect breaking changes, and it needed to support company-wide standards as well as service-specific standards. We use Smithy’s validation system to automatically enforce API standards, and service teams often create their own service-specific rules to keep their own internal consistency.
We needed built-in input validation constraints so that they're standard across services and clients (e.g., length, range, pattern, etc). We didn't want to rely on third-party extensions to provide this feature since validating inputs is important. AWS uses internal service frameworks that enforce these constraints and are compatible with Smithy models. We're working to create open source service frameworks for Smithy as well.
We also wanted to support various serialization formats so that clients work with all of our existing services spread across JSON, XML, query strings, RPC, and HTTP APIs, but we also wanted to be able to evolve our serialization formats in the future as new technology comes along. That's why Smithy is protocol agnostic (like gRPC actually). The serialization format is an implementation detail. Smithy has some support for MQTT as well.
And finally, we need our code generators to be really flexible to support service customizations. There's quite a few customizations across AWS services, and we needed a way to inject custom code generation logic in various parts of our generators.
Smithy is still in heavy development, and we're working on building out more of the tooling so it can be used easily outside of AWS SDKs too, including client and server code generation.
Thanks for the thorough response! This helps me understand the motivations behind Smithy. I’m going to dig into the project and keep an eye on it as the tooling develops.
> Smithy's resource modeling helps us here too because it allows AWS service teams, as they adopt Smithy, to essentially automatically support CloudFormation resource schemas.
There's not a trait for it yet, but Smithy is designed to be auth agnostic so new auth traits can be added by anyone. There's a meta-trait called authDefinition[0] that you can apply to your own trait to indicate that it's an auth trait. With that your trait would show up anywhere else auth traits are found in the Smithy tooling. We're designing the code generators to be extensible enough that you could then fairly easily implement just the necessary bits.
[0] https://developers.google.com/protocol-buffers