Combining XAML and DirectX

Since early on in the development of Windows 8, we’ve been thinking about ways to combine UI, such as the rich set of controls and interactivity provided by XAML, and graphics, like the low-level, high-performance rendering of DirectX.

The feedback you gave for the Developer Preview through the Dev Center forums and other channels helped us focus on the set of related scenarios that developers really wanted to build. We saw some common themes: many of you wanted to either add DirectX graphics to what could otherwise be called a XAML app, or easily add Metro style UI to a DirectX app.

Good news! With the release of the Consumer Preview, you no longer have to draw a hard distinction between a XAML app and a DirectX app. You can now tap into the strengths of both XAML as a rich UI platform and DirectX as a high-performance rendering solution within the same app, using a familiar XAML model.

Now that you can combine these, you really get the best of both worlds. This opens up a wide range of possibilities – some of the key scenarios you told us about included things like:

  • Image processing, creativity, and design apps that blend UI and graphics
  • Large-scale map and document viewers, that mix huge DirectX graphics and some UI
  • Games and simulations with full-screen, high-performance DirectX graphics and minimal overlaid UI

where the combination of XAML and DirectX can make developers more productive and apps richer and faster.

Metro style UI in Windows 8

XAML – in parallel with HTML/JavaScript – is a UI toolkit for all types of apps in Windows 8, providing constructs for interactivity, Metro style controls and animations, and rich functionality like accessibility support, databinding, media, and HTML hosting. You also benefit from design-time support in Visual Studio and Expression Blend, all using your choice of programming languages.

One area of continual investment for the engineering team that spans across all these areas in both XAML and HTML/JavaScript is performance: we’ve put a lot of effort into this in Windows 8, making it easier than ever for you to build fast and fluid Metro style apps. In the graphics area in particular, we’ve continued to improve rendering and composition performance, worked to use GPU hardware wherever possible, and even made it so that common types of animations run independently off the UI thread to ensure they remain smooth regardless of any work your app is doing on the CPU. We’ve worked toward ensuring that everything the UI frameworks do, they do well — fast, fluid, and with lower power requirements. When using a UI framework, you’re already using the power of DirectX behind the scenes; in many cases this provides everything you need to achieve great graphics performance.

DirectX in Windows 8

Still, there are definitely times when you need the raw immediate-mode rendering performance and direct access to graphics devices that DirectX provides. Starting with Windows 8, DirectX SDK components are now included as part of the primary Windows SDK, and DirectX contains a host of new features such as a unified 3D API, improved 2D graphics and text performance with Direct2D, a rich image processing pipeline, and improved printing support.

Everything in Windows 8 is optimized for and built around DirectX, from the developer platforms to the OS to hardware design. This way, you can use DirectX to achieve the highest performance rendering in Windows 8. The major tradeoff is that pure DirectX solutions can be quite complex to create and maintain, and (unless you’re using a third-party wrapper) you must implement them in C++.

This is where DirectX and XAML interop comes into play: you can now build your XAML UI using C#, Visual Basic or C++ and include C++ DirectX components. You may also be wondering about HTML/JavaScript and DirectX – as mentioned earlier we’ve done a lot to bake the power of DirectX into the UI platforms, but for combining UI and native DirectX graphics we’ve focused on XAML for Windows 8.

Design goals

With the above in mind, we developed a few principal goals we wanted to meet when enabling the combination of DirectX and XAML:

1. Performance

  • Support low-latency input and interactivity
  • Allow incremental redrawing of both XAML and DirectX content
  • Minimize overhead from using XAML

Chiefly, we wanted to avoid imposing a “XAML tax”: if your app is primarily DirectX, using XAML should be an easy choice if you need Metro style UI. We know this isn’t important just to you: we wanted to help ensure your customers see great performance and long battery life from your app.

2. Flexibility

  • Enable all functionality of both XAML and DirectX 11

The result is that you can design and develop a XAML interface in Blend and Visual Studio, and combine it with any DirectX content. We wanted to make sure we weren’t limiting the power of either XAML or DirectX — just letting you combine them.

3. Integration

  • Ensure smooth integration of XAML and DirectX

You can combine DirectX graphics and XAML UI in the same app using familiar XAML concepts.

After breaking down the set of scenarios listed earlier, we found they fell into three broad categories:

  1. Combining small islands of DirectX into a primarily XAML UI
  2. Combining large scalable DirectX surfaces with XAML UI
  3. Overlaying Metro style XAML UI onto a primarily DirectX app

In the end, we found that a one-size-fits-all approach didn’t work well across all these categories. So, we added three XAML types that each targets one of these key types of scenario. We believe this provides a high degree of flexibility for you to choose the option(s) best suited to the requirements of your app. These options aren’t exclusive — you could even use all three approaches for different components in the same app.

Scenario #1 – SurfaceImageSource: islands of DirectX in a XAML UI

As the first option for combining DirectX and XAML, you can use the new SurfaceImageSource XAML type to add areas of DirectX content to a Metro style XAML app. An instance of this type provides you with a shared DirectX surface you can draw to using all the functionality of DirectX, then composes the surface into the rest of your XAML app’s interface.

This option is particularly useful when you want to use DirectX to draw specific components that are part of a larger XAML UI. For example, you might want to use SurfaceImageSource when applying complex image effects to a photo, or when creating high-performance 3D data visualizations.

If you’ve ever used a XAML ImageSource, the usage will be quite familiar: you can set a SurfaceImageSource as the source of an ImageBrush and then use it to paint nearly any XAML element (or elements — brushes are reusable!). For example, you could use a SurfaceImageSource to fill a XAML Rectangle:

SurfaceImageSource^ surfaceImageSource =  
ref new SurfaceImageSource(rectangle1->Width, rectangle1->Height, true);
ImageBrush^ brush = ref new ImageBrush();
brush->ImageSource = surfaceImageSource;
rectangle1->Fill = brush;

You can then use DirectX to draw the content of the Rectangle, and update it as often as you need to. The best part is that this surface is seamlessly integrated with the XAML composition model: properties applied to that Rectangle also affect the content you draw with DirectX. You can easily overlap elements, and apply render transforms, projections, z-index, or opacity, all in XAML, without any of the airspace issues often experienced when mixing rendering technologies:

A diagram showing a XAML app that mixes a DirectX surface and XAML content. The DirectX surface is rotated and composed with other XAML content that is overlaid on top of the surface.

Rectangular DirectX surface composed with XAML content

If you’re thinking this seems conceptually similar to the D3DImage type in WPF, then you’re right! However, we listened to your feedback on D3DImage and worked to make SurfaceImageSource both more powerful and easier to use. In particular, SurfaceImageSource: supports DirectX 11, including Direct2D; removes many of the limitations that D3DImage had (for instance, SurfaceImageSource works over a Remote Desktop connection); improves performance; and manages the surfaces for you — no more locking and unlocking, or manually creating and managing buffers! A SurfaceImageSource can even support surfaces larger than your Direct3D device’s maximum texture size.

The first thing you might notice when creating a SurfaceImageSource is that at a glance it doesn’t seem to expose any new methods. The implementation is in fact exposed via a backing native interface; this is because it needs to directly reference DirectX APIs, which are available only in C++. This means you must use C++ when calling DirectX APIs, either directly in a C++ XAML app or in a separate C++ WinRT component that you can then include in any C# or Visual Basic app. Once you include the required header file:

#include "windows.ui.xaml.media.dxinterop.h"

Then you can query for the ISurfaceImageSourceNative interface, which gives access to the three methods you need to get started:

interface ISurfaceImageSourceNative: IUnknown
{
SetDevice(IDXGIDevice *pDevice);
BeginDraw(
RECT updateRect,
IDXGISurface** pSurface,
POINT* offset
);

EndDraw();
};

BeginDraw() takes an updateRect parameter that enables you to constrain the area you want to update; this allows you to improve performance by making only smaller incremental updates to the surface. The method also returns a surface and a point specifying an offset where you should begin drawing: even if you’re updating the whole surface, you should still make sure to check this offset, because your surface may be part of a larger one the framework maintains internally as an optimization. From here you can create and set a DXGI Device, and then start drawing with DirectX! The DirectX and XAML interop Dev Center topic provides a good overview of the details.

Scenario #2 – VirtualSurfaceImageSource: large-scale DirectX content + XAML UI

VirtualSurfaceImageSource extends SurfaceImageSource to better support huge surfaces. Its foundation lies in the concept of virtualization — that is, drawing areas of a surface only as they become visible on the screen.

We designed this type to be used when displaying complex content that is larger than the screen, especially when it can be panned or zoomed. Scenarios where this might apply include anything from a document viewer to a medical imaging app — apps with content that could potentially be many times larger than the maximum resolution of an average monitor. A map control is another good example: using a VirtualSurfaceImageSource, you could implement a large vector-based map with DirectX and let the system do the work of tracking the area that’s on-screen at any given time. You can also overlay on the map XAML elements such as pushpins, and easily keep their positions in sync with the DirectX content.

VirtualSurfaceImageSource implements this virtualization using tiling and a callback model. After you create your full-size VirtualSurfaceImageSource, your DirectX content gets split up into a series of rectangular areas behind the scenes; the framework automatically keeps track of the visible rects at any given point (plus potentially some extra just out of view, to keep panning smooth), and invokes a callback that your app implements to provide the content for newly visible regions when needed. This works especially well when you place your VirtualSurfaceImageSource content in a ScrollViewer to enable smooth panning with touch or a mouse.

A large DirectX surface with area that is currently visible framed by a screen. Blank regions outside the screen represent parts of the surface that have not been drawn yet because they aren’t yet visible on the screen.

Large surface split into rectangular tiles

This approach isn’t just easier to implement than tracking coordinates yourself: it can provide a large boost to your app’s performance. Because the system caches your content internally, after you draw the content of a region it’s kept in memory and will automatically be reused by the GPU even if it’s panned out of view and then back again — no redrawing required! If you want to update any region you’ve already drawn, you can simply call BeginDraw() as you would with a SurfaceImageSource. As a further performance optimization, the system automatically recycles memory used for old regions that are out of view after a certain point, helping to bound memory usage and ensure your app will perform well across a range of form factors and hardware profiles even with very large surfaces. This performance benefit becomes especially apparent on portable devices such as tablets.

Similar to using SurfaceImageSource, you can query for the backing native interface, in this case IVirtualSurfaceImageSourceNative, to access the required methods. You can check out the DirectX and XAML interop Dev Center article or the Direct2D magazine app sample for details.

Scenario #3 – SwapChainBackgroundPanel: DirectX with a XAML overlay

The third and final option for combining XAML and DirectX content, SwapChainBackgroundPanel is a XAML element for developing full-screen, primarily DirectX-focused apps. Unlike SurfaceImageSource and VirtualSurfaceImageSource, SwapChainBackgroundPanel is a XAML element rather than an ImageSource; it inherits from the XAML Grid layout panel and enables apps to create and fully control a full-screen DirectX swap chain, on top of which all other XAML content is overlaid:

Illustration of composition model of SwapChainBackgroundPanel, where DirectX content always sits behind a XAML overlay

A SwapChainBackgroundPanel sits behind overlaid XAML content

We think this is definitely the best option for games, and for other apps where most of your graphics are DirectX; you get great performance and an update model close to a pure DirectX app, and you can use XAML at the same time to easily create a heads-up display interface and databind it to your game model, add interactive controls, and include Metro style elements such as AppBars.

To achieve this, we impose a couple restrictions on the SwapChainBackgroundPanel: it must be the root XAML element of your app, and a few inherited XAML properties, such as Opacity and RenderTransform, won’t have any effect on it (though you can of course apply the equivalent DirectX operations to the swap chain content to achieve these effects). This allows you to make a tradeoff between the close integration with the XAML framework’s composition model that you get with SurfaceImageSource, and the lowest-latency interaction and extra control over presentation timing possible when you directly manage the swap chain.

The backing native interface in this case has only a single method:

interface ISwapChainBackgroundPanelNative: IUnknown
{
SetSwapChain(IDXGISwapChain *pSwapChain);
};

You can start presenting your content as soon as you call SetSwapChain() to associate your swap chain with the panel. At that point SwapChainBackgroundPanel also becomes hit-testable and you handle input using normal XAML input events. Any child XAML elements you add to the panel are displayed on top of your swap chain.

The interop Dev Center topic has more info on SwapChainBackgroundPanel usage, and you can take a look at the source for a sample DirectX shooting game built using SwapChainBackgroundPanel that shows the end-to-end implementation of a Direct3D game with a XAML interface:

A screenshot of a DirectX and XAML first-person shooter game. The DirectX surface contains walls and targets overlaid with a XAML UI containing a menu and an AppBar control.

A game built with DirectX and XAML

In closing

We’re excited about the new possibilities that open up when you combine the rich UI support and design-time productivity of XAML with the high-performance graphics power of DirectX. We hope you find these three new options for building Metro style apps useful when you create compelling apps for your customers.

– Jesse Bishop

    Program Manager, Windows


Windows 8 app developer blog

Leave a Reply