Dispatch

This chapter attempts to document the Vulkan dispatch infrastructure in the Mesa Vulkan runtime. There are a lot of moving pieces here but the end result has proven quite effective for implementing all the various Vulkan API requirements.

Extension tables

The Vulkan runtime defines two extension table structures, one for instance extensions and one for device extensions which contain a Boolean per extension. The device table looks like this:

#define VK_DEVICE_EXTENSION_COUNT 238

struct vk_device_extension_table {
   union {
      bool extensions[VK_DEVICE_EXTENSION_COUNT];
      struct {
         bool KHR_8bit_storage;
         bool KHR_16bit_storage;
         bool KHR_acceleration_structure;
         bool KHR_bind_memory2;
         ...
      };
   };
};

The instance extension table is similar except that it includes the instance level extensions. Both tables are actually unions so that you can access the table either by name or as an array. Accessing by name is typically better for human-written code which needs to query for specific enabled extensions or declare a table of which extensions a driver supports. The array form is convenient for more automatic code which wants to iterate over the table.

These tables are are generated automatically using a bit of python code that parses the vk.xml from the Vulkan-Docs repo, enumerates the extensions, sorts them by instance vs. device and generates the table. Generating it from XML means that we never have to manually maintain any of these data structures; they get automatically updated when someone imports a new version of vk.xml. We also generates a matching pair of tables of VkExtensionProperties. This makes it easy to implement vkEnumerate*ExtensionProperties() with a simple loop that walks a table of supported extensions and copies the VkExtensionProperties for each enabled entry. Similarly, we can have a loop in vkCreateInstance() or vkCreateDevice() which takes the ppEnabledExtensionNames and fills out the table with all enabled extensions.

Entrypoint and dispatch tables

Entrypoint tables contain a function pointer for every Vulkan entrypoint within a particular scope. There are separate tables for instance, physical device, and device-level functionality. The device entrypoint table looks like this:

struct vk_device_entrypoint_table {
   PFN_vkGetDeviceProcAddr GetDeviceProcAddr;
   PFN_vkDestroyDevice DestroyDevice;
   PFN_vkGetDeviceQueue GetDeviceQueue;
   PFN_vkQueueSubmit QueueSubmit;
   ...
#ifdef VK_USE_PLATFORM_WIN32_KHR
   PFN_vkGetSemaphoreWin32HandleKHR GetSemaphoreWin32HandleKHR;
#else
   PFN_vkVoidFunction GetSemaphoreWin32HandleKHR;
# endif
   ...
};

Every entry that requires some sort of platform define is wrapped in an #ifdef and declared as the actual function pointer type if the platform define is set and declared as a void function otherwise. This ensures that the layout of the structure doesn’t change based on preprocessor symbols but anyone who has the platform defines set gets the real prototype and anyone who doesn’t can use the table without needing to pull in all the platform headers.

Dispatch tables are similar to entrypoint tables except that they’re deduplicated so that aliased entrypoints have only one entry in the table. The device dispatch table looks like this:

struct vk_device_dispatch_table {
    PFN_vkGetDeviceProcAddr GetDeviceProcAddr;
    PFN_vkDestroyDevice DestroyDevice;
    PFN_vkGetDeviceQueue GetDeviceQueue;
    PFN_vkQueueSubmit QueueSubmit;
    ...
    union {
        PFN_vkResetQueryPool ResetQueryPool;
        PFN_vkResetQueryPoolEXT ResetQueryPoolEXT;
    };
    ...
};

In order to allow code to use any of the aliases for a given entrypoint, such entrypoints are wrapped in a union. This is important because we need to be able to add new aliases potentially at any Vulkan release and we want to do so without having to update all the driver code which uses one of the newly aliased entrypoints. We could require that everyone use the first name an entrypoint ever has but that gets weird if, for instance, it’s introduced in an EXT extension and some driver only ever implements the KHR or core version of the feature. It’s easier for everyone if we make all the entrypoint names work.

An entrypoint table can be converted to a dispatch table by compacting it with one of the vk_*_dispatch_table_from_entrypoints() family of functions:

void vk_instance_dispatch_table_from_entrypoints(
    struct vk_instance_dispatch_table *dispatch_table,
    const struct vk_instance_entrypoint_table *entrypoint_table,
    bool overwrite);

void vk_physical_device_dispatch_table_from_entrypoints(
    struct vk_physical_device_dispatch_table *dispatch_table,
    const struct vk_physical_device_entrypoint_table *entrypoint_table,
    bool overwrite);

void vk_device_dispatch_table_from_entrypoints(
    struct vk_device_dispatch_table *dispatch_table,
    const struct vk_device_entrypoint_table *entrypoint_table,
    bool overwrite);

Generating driver dispatch tables

Entrypoint tables can be easily auto-generated for your driver. Simply put the following in the driver’s meson.build, modified as necessary:

drv_entrypoints = custom_target(
  'drv_entrypoints',
  input : [vk_entrypoints_gen, vk_api_xml],
  output : ['drv_entrypoints.h', 'drv_entrypoints.c'],
  command : [
    prog_python, '@INPUT0@', '--xml', '@INPUT1@', '--proto', '--weak',
    '--out-h', '@OUTPUT0@', '--out-c', '@OUTPUT1@', '--prefix', 'drv',
    '--beta', with_vulkan_beta.to_string(),
  ],
  depend_files : vk_entrypoints_gen_depend_files,
)

The generated drv_entrypoints.h fill will contain prototypes for every Vulkan entrypoint, prefixed with what you passed to --prefix above. For instance, if you set --prefix drv and the entrypoint name is vkCreateDevice(), the driver entrypoint will be named drv_CreateDevice(). The --prefix flag can be specified multiple times if you want more than one table. It also generates an entrypoint table for each prefix and each dispatch level (instance, physical device, and device) which is populated using the driver’s functions. Thanks to our use of weak function pointers (or something roughly equivalent for MSVC), any entrypoints which are not implemented will automatically show up as NULL entries in the table rather than resulting in linking errors.

The above generates entrypoint tables because, thanks to aliasing and the C rules around const struct declarations, it’s not practical to generate a dispatch table directly. Before they can be passed into the relevant vk_*_init() function, the entrypoint table will have to be converted to a dispatch table. The typical pattern for this inside a driver looks something like this:

struct vk_instance_dispatch_table dispatch_table;
vk_instance_dispatch_table_from_entrypoints(
   &dispatch_table, &anv_instance_entrypoints, true);
vk_instance_dispatch_table_from_entrypoints(
   &dispatch_table, &wsi_instance_entrypoints, false);

result = vk_instance_init(&instance->vk, &instance_extensions,
                          &dispatch_table, pCreateInfo, pAllocator);
if (result != VK_SUCCESS) {
   vk_free(pAllocator, instance);
   return result;
}

The vk_*_dispatch_table_from_entrypoints() functions are designed so that they can be layered like this. In this case, it starts with the instance entrypoints from the Intel Vulkan driver and then adds in the WSI entrypoints. If there are any entrypoints duplicated between the two, the first one to define the entrypoint wins.

Common Vulkan entrypoints

For the Vulkan runtime itself, there is a dispatch table with the vk_common prefix used to provide common implementations of various entrypoints. This entrypoint table is added last as part of vk_*_init() so that the driver implementation will always be used, if there is one.

This is used to implement a bunch of things on behalf of the driver. The most common case is whenever there are vkFoo() and vkFoo2() entrypoints. We provide wrappers for nearly all of these that implement vkFoo() in terms of vkFoo2() so a driver can switch to the new one and throw the old one away. For instance, vk_common_BindBufferMemory() looks like this:

VKAPI_ATTR VkResult VKAPI_CALL
vk_common_BindBufferMemory(VkDevice _device,
                           VkBuffer buffer,
                           VkDeviceMemory memory,
                           VkDeviceSize memoryOffset)
{
   VK_FROM_HANDLE(vk_device, device, _device);

   VkBindBufferMemoryInfo bind = {
      .sType         = VK_STRUCTURE_TYPE_BIND_BUFFER_MEMORY_INFO,
      .buffer        = buffer,
      .memory        = memory,
      .memoryOffset  = memoryOffset,
   };

   return device->dispatch_table.BindBufferMemory2(_device, 1, &bind);
}

There are, of course, far more complicated cases of implementing vkFoo() in terms of vkFoo2() such as the vk_common_QueueSubmit() implementation. We also implement far less trivial functionality as vk_common_* entrypoints. For instance, we have full implementations of VkFence, VkSemaphore, and vkQueueSubmit2().

Entrypoint lookup

Implementing vkGet*ProcAddr() is quite complicated because of the Vulkan 1.2 rules around exactly when they have to return NULL. When a client calls vkGet*ProcAddr(), we go through a three step process resolve the function pointer:

  1. A static (generated at compile time) hash table is used to map the entrypoint name to an index into the corresponding entry point table.

  2. Optionally, the index is passed to an auto-generated function that checks against the enabled core API version and extensions. We use an index into the entrypoint table, not the dispatch table, because the rules for when an entrypoint should be exposed are per-entrypoint. For instance, vkBindImageMemory2 is available on Vulkan 1.1 and later but vkBindImageMemory2KHR is available if VK_KHR_bind_memory2 is enabled.

  3. A compaction table is used to map from the entrypoint table index to the dispatch table index and the function is finally fetched from the dispatch table.

All of this is encapsulated within the vk_*_dispatch_table_get() and vk_*_dispatch_table_get_if_supported() families of functions. The _if_supported versions take a core version and one or more extension tables. The driver has to provide vk_icdGet*ProcAddr() entrypoints which wrap these functions because those have to be exposed as actual symbols from the .so or .dll as part of the loader interface. It also has to provide its own drv_GetInstanceProcAddr() because it needs to pass the supported instance extension table to vk_instance_get_proc_addr(). The runtime will provide vk_common_GetDeviceProcAddr() implementations.

Populating layer or client dispatch tables

The entrypoint and dispatch tables actually live in src/vulkan/util, not src/vulkan/runtime so they can be used by layers and clients (such as Zink) as well as the runtime. Layers and clients may wish to populate dispatch tables from an underlying Vulkan implementation. This can be done via the vk_*_dispatch_table_load() family of functions:

void
vk_instance_dispatch_table_load(struct vk_instance_dispatch_table *table,
                                PFN_vkGetInstanceProcAddr gpa,
                                VkInstance instance);
void
vk_physical_device_dispatch_table_load(struct vk_physical_device_dispatch_table *table,
                                       PFN_vkGetInstanceProcAddr gpa,
                                       VkInstance instance);
void
vk_device_dispatch_table_load(struct vk_device_dispatch_table *table,
                              PFN_vkGetDeviceProcAddr gpa,
                              VkDevice device);

These call the given vkGet*ProcAddr function to populate the dispatch table. For aliased entrypoints, it will try each variant in succession to ensure that the dispatch table entry gets populated no matter which version of the feature you have enabled.